Hello everyone!

I've recently got an idea for a Database as a Service I'd like to create. The only issue is - I don't know how to create or host a database!

I've only ever used Mongoose/mLab with Javascript, and a minimal amount of Postgres with Python.

If I'm looking to create a database that will, eventually, be able to store images, songs, and videos, where should I start my homework?

I can create the backend and the frontend with no issue - just stuck on this part here. If it's of any relevance, I most frequently use the MERN stack.

Hey Tildes, I got a job directly supervising a small team of 4 software developers. I'm very excited at the prospect and would like to put my best foot forward. To that end, I would like to have a discussion around a few topics. Feel free to expand the scope if you believe the conversation would be beneficial. I'm sure I won't be the last person to be in this position. I've done research, read, and watched videos regarding several of these questions; however, since Tilde prioritizes high-quality discussion, I thought it would be a fun opportunity to chat with others about these topics.

• As a member of a software development team, what are things that your supervisor has done that has had the greatest (a) positive and (b) negative impact?
• Supervisors, when you joined your new team, what was your methodology for reviewing the team, projects, and processes? What was the scenarios behind your review and the outcome? What would you do differently?

I'm currently looking for a new monitor and I have the ASUS MG279Q, the ASUS PG248Q and the ASUS MG278Q. Now, my setup is not the highest end, but decent with an i7 and a 1060 3GB, and there are three concerns I have currently not found an answer for:

1. The (potential) difference in quality between the MG279Q's IPS panel and the MG278's TN panel
2. And the delay difference, the TN panel having a 1ms response time and the IPS' 4ms
3. If my setup can even handle 1440p/144Hz (I don't need to play on the highest settings, nor do I need to reach those 144 FPS), in which case I would tend more towards the PG248Q

I'd love to upgrade towards 1440p as the screen real estate would be good for working (which I do a lot on the PC) and I would, I could later upgrade my GPU if the performance in games isn't satisfactory. I think my setup wouldn't have any issue handling day to day tasks and if need be I can play on lower resolutions or lower graphic settings. Also I wonder how large the difference between the IPS and TN panel is and if it's noticeable, particularly with colours.

Does anyone of you have experience with the subject or with the monitors named in particular?

Preface

In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer productivity, and the need to pay down on that debt. Today I would like to touch on a practical example that I'd encountered in a production environment.

Background

Before we can discuss the refactor itself, it's important to be on the same page regarding the technologies being used. In my case, I work with PHP utilizing a proprietary back-end framework and MongoDB as our database.

PHP is a server-side scripting language. Like many scripting languages, it's loosely typed. This has some benefits and drawbacks.

MongoDB is a document-oriented database. By default it's schema-less, allowing you to make any changes at will without an update to schema. This can blend pretty well with the loose typing of PHP. Each document is represented using a JSON-like structure and is stored in something called a "collection". For those of you accustomed to using relational database, a "collection" is analogous to a table, each document is a row, and each field in the document is a column. A typical query in the MongoDB shell would look something like this:

db.users.findOne({
    username: "Emerald_Knight"
});

The framework itself has some framework-specific objects that are held in global references. This makes them easily accessible, but naturally littering your code with a bunch of globals is both error-prone and an eyesore.

Unexpected Spaghetti

In my code base are a number of different objects that are designed to handle basic CRUD-like operations on their associated database entries. Some of these objects hold references to other objects, so naturally there is some data validation that occurs to ensure that the references are both valid and authorized. Pretty typical stuff.

What I noticed, however, is that the collection names for these database entries were littered throughout my code. This isn't necessarily a bad thing, except there were some use cases that came to mind: what if it turned out that my naming for one or more of these collections wasn't ideal? What if I wanted to change a collection name for the sake of easier management on the database end? What if I have a tendency to forget the name of a database collection and constantly have to look it up? What if I make a typo of all things? On top of that, the framework's database object was stored in a global variable.

These seemingly minor sources of technical debt end up adding up over time and could cause some serious problems in the worst case. I've had breaking bugs make their way passed QA in the past, after all.

Exchanging Spaghetti for Some Light Lasagna

The problem could be characterized simply: there were scoping problems and too many references to what were essentially magic strings. The solution, then, was to move the database object reference from global to local scope within the application code and to eliminate the problem of magic strings. Additionally, it's a good idea to avoid polluting the namespace with an over-reliance on constants, and using those constants for database calls can also become unsightly and difficult to follow as those constants could end up being generally disconnected from the objects they're associated with.

There turned out to be a nice, object-oriented, very PHP-like solution to this problem: a so-called "magic method" named "__call". This method is invoked whenever an "inaccessible" method is called on the object. Using this method, a database command executed on a non-database object could pass the command to the database object itself. If this logic were placed within an abstract class, the collection could then be specified simply as a configuration option in the inheriting class.

This is what such a solution could look like:

<?php

abstract class MyBaseObject {

    protected $db = null;
    protected $collection_name = null;

    public function __construct() {
        global $db;
        
        $this->db = $db;
    }

    public function __call($method_name, $args) {
        if(method_exists($this->db, $method_name)) {
            return $this->executeDatabaseCommand($method_name, $args);
        }

        throw new Exception(__CLASS__ . ': Method "' . $method_name . '" does not exist.');
    }

    public function executeDatabaseCommand($command, $args) {
        $collection = $this->collection_name;
        $db_collection = $this->db->$collection;

        return call_user_func_array(array($db_collection, $command), $args);
    }
}

class UserManager extends MyBaseObject {
    protected $collection_name = 'users';

    public function __construct() {
        parent::__construct();
    }
}

$user_manager = new UserManager();
$my_user = $user_manager->findOne(array('username'=>'Emerald_Knight'));

?>

This solution utilizes a single parent object which transforms a global database object reference into a local one, eliminating the scope issue. The collection name is specified as a class property of the inheriting object and only used in a single place in the parent object, eliminating the magic string and namespace polluting issues. Any time you perform queries on users, you do so by using the UserManager class, which guarantees that you will always know that your queries are being performed on the objects that you intend. And finally, if the collection name for an object class ever needs to be updated, it's a simple matter of modifying the single instance of the class property $collection_name, rather than tracking down some disconnected constant.

Limitations

This, of course, doesn't solve all of the existing problems. After all, executing the database queries for one object directly from another is still pretty bad practice, violating the principle of separation of concerns. Instead, those queries should generally be encapsulated within object methods and the objects themselves given primary responsibility in handling associated data. It's also incredibly easy to inadvertently override a database method, e.g. defining a findOne() method on UserManager, so there's still some mindfulness required on the part of the programmer.

Still, given the previous alternative, this is a pretty major improvement, especially for an initial refactor.

Final Thoughts

As always, technical debt is both necessary and inevitable. After all, in exchange for not taking the excess time and considering structuring my code this way in the beginning, I had greater initial velocity to get the project off of the ground. What's important is continually reviewing your code as you're building on top of it so that you can identify bottlenecks as they begin to strain your efficiency, and getting those bottlenecks out of the way.

In other words, even though technical debt is often necessary and is certainly inevitable, it's important to pay down on some of that debt once it starts getting expensive!

[seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly]

I've had some experience in this realm for a while now, but I'm having a little trouble with one issue in particular. Before I divulge, I'll present my thoughts on best practice and and what I've been able to implement:

• Terraform everything (in accordance to terragrunt's "style guide" i.e. organization)
  THIS IS A BIG ONE: for the jenkins master task, make sure to use the following args to make sure jenkins jobs aren't super slow as hell to start:

-Djava.awt.headless=true -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

THIS IS A GAME CHANGER (more-so on k8s clusters when the ecs plugin isn't used... hint, it's shit).

• Create an EFS (in a separate terraform module) and mount it to the jenkins ECS cluster at /var/jenkins_home. Makes jenkins much more reliable through outages and easier to upgrade.
• Run a logging agent (via docker container) like logspout or newrelic or whatever IN USER_DATA and not as a task - that way you get logs if there are issues during user_data/cloud_init... this I'm actually not sure about. Running a container outside the context of an ECS task means the ECS agent can't really track it and allocate mem/cpu properly... but it does help with user_data triage.
• Use pipelines and git plugins to drive jobs. All jenkins jobs should be in source control!
• Make sure you setup docker cleanup jobs on DAY 1! If you hace limited access to your cluster and you run out of disk due to docker cache, networks, volumes, etc... you're screwed till the admin ssh's in and runs a prune. Get a docker system prune going or the equivalent for each docker resource with appropriate filters... i.e. filter for anything older than a few days and is dangling.
• Use Jenkins Global Libraries to make Jenkinsfiles cleaner (I always just use vars instead of groovy/java style packages because it's easier and less ugly)
  Jenkinsfiles should mostly call other bash files, make files, python scripts to generate and load prop files, etc. The less logic you put in a Jenkinsfile (which is just modified groovy) the better. String interpolation, among other things, is a fuckery that we don't have time to triage.
• (out-of-scope) Move to using k8s/EKS instead of ECS asap because the ECS plugin for jenkins is absolute shit and it doesn't use priority correctly (sorry whoever developed it and... oh wait abandoned it and hasn't merged anything for years... for for real it's cool, just give admin to someone else).
• (cultural) Stop calling them slaves. "Hey @eng, we're rotating slaves due to some cache issues. If you have been affected by race conditions in that past, our new update and slave rotation should fix that. Our update may have killed your job that was running on an old slave, just wait a few and the new slaves will be ready" <--This just doesn't look good.
  Hope that was some good stuff for you guys. Maybe I'm preaching to the choir, but I've seen some pretty shit jenkins setups.

NOW FOR MY QUESTION!

Has ANYONE actually been able to setup a proper jenkins user on ECS that actually works for both a master and ephemeral jenkins-agents so that they can mount and use the docker.sock for builds without hitting permission issues? I'm talking using the ecs plugin and mounting docker.sock via that.

I have always resorted to running jenkins master and agents as root, which means you have to chmod files (super expensive time and cpu for services with tons of files). Running microservices as root is obviously bad practice, and chmod-ing a zilliion files is shit for docker cache and time... so I want to get jenkins users able to utilize the docker.sock. THIS IS SPECIFICALLY FOR THE AWS ECS AMI! I don't care about debian or old versions of docker where you could use DOCKER_OPTS. That doesn't work on the AWS Linux image.

Thanks! And happy Friday!

I have been interested in making the gradual career change to software development from my current humanities field. This stems from a handful of different places. Of course the pay and flexibility are strong drivers but I like the idea of a field that is somewhat of a creative expression; one where you can manifest your knowledge and experience into something tangible.

I have no experience with programming other than SQL use in ArcGIS and am hoping to gain some knowledge about the field; so anything would be helpful. Whether what to expect from this line of work, where someone with no experience should look to get started and what to expect, personal journeys, etc.

Cheers!

In most of my programming, I try and remain professional, and do things in a readable, maintainable way, that doesn't involve pushing the language to breaking point.

But, occasionally, I give myself free reign. What if you didn't care about the programmer who came after you? What if you didn't care about what a programmer should do in a certain circumstance?

For myself, over the years I've written and rewritten a C library, I like to call CNoEvil. Here's a little taste of what you could do:

#define EVIL_IO
#define EVIL_COROUTINE
#include "evil.h"

proc(example, int)
  static int i = 0;
  coroutine();
  While 1 then
    co_return(++i);
  end
  co_end();
  return i;
end

Main then
  displayln(example());
  displayln(example());
end

(And yes, that compiles. Without warnings. Even with -Wpedantic.)

So... Here's the challenge:

Ignoring the rules, and best practices... How can you take your favourite programming language... And make it completely unrecogniseable?

(Might be best to choose a language with macros, like Nim, Rust, any of the Lisps. Though, you can still do some impressively awful things in Java or Python, thanks to overloading inbuilt classes.)

Challenge Ideas:

• Make Python look like C
• Make Java look like Python
• Make anything look like BrainFuck

I don’t know how to really explain my fascination with programming, but I’ll try. To somebody who does it, it’s the most interesting thing in the world. It’s a game much more involved than chess, a game where you can make up your own rules and where the end result is whatever you can make of it. - Linus Torvalds

I am looking for any resource that could tell me how much knowledge and training is needed to go from a beginner to expert, in let's say application software development for example. Or in artificial intelligence.

If there isn't any one source, are there any general type of sources I can use to piece together one mega-source?

Hey y'all, first time actually posting something here! Just curious what editor people use, whether its for coding, writing, or just the occasional note, whatever. I've gone through most of the well-known ones (vim, emacs, atom, vs code for starters), but only ever really messed around with vim enough to like it, and I've also been trying out gedit for the last little while and really liking it, but I'm curious to see what other people use!

Here’s a problem I had to solve at work this week that I enjoyed solving. I think it’s a good programming challenge that will test if you really grok XML.

Your input is some XML such as this:

<DOC>
<TEXT PARTNO="000">
<TAG ID="3">This</TAG> is <TAG ID="0">some *JUNK* data</TAG> .
</TEXT>
<TEXT PARTNO="001">
*FOO* Sometimes <TAG ID="1">tags in <TAG ID="0">the data</TAG> are nested</TAG> .
</TEXT>
<TEXT PARTNO="002">
In addition to <TAG ID="1">nested tags</TAG> , sometimes there is also <TAG ID="2">junk</TAG> we need to ignore .
</TEXT>
<TEXT PARTNO="003">*BAR*-1
<TAG ID="2">Junk</TAG> is marked by uppercase characters between asterisks and can also optionally be followed by a dash and then one or more digits . *JUNK*-123
</TEXT>
<TEXT PARTNO="004">
Note that <TAG ID="4">*this*</TAG> is just emphasized . It's not <TAG ID="2">junk</TAG> !
</TEXT>
</DOC>

The above XML has so-called in-line textual annotations because the XML <TAG> elements are embedded within the document text itself.

Your goal is to convert the in-line XML annotations to so-called stand-off annotations where the text is separated from the annotations and the annotations refer to the text via slicing into the text as a character array with starting and ending character offsets. While in-line annotations are more human-readable, stand-off annotations are equally machine-readable, and stand-off annotations can be modified without changing the document content itself (the text is immutable).

The challenge, then, is to convert to a stand-off JSON format that includes the plain-text of the document and the XML tag annotations grouped by their tag element IDs. In order to preserve the annotation information from the original XML, you must keep track of each <TAG>’s starting and ending character offset within the plain-text of the document. The plain-text is defined as the character data in the XML document ignoring any junk. We’ll define junk as one or more uppercase ASCII characters [A-Z]+ between two *, and optionally a trailing dash - followed by any number of digits [0-9]+.

Here is the desired JSON output for the above example to test your solution:

{
  "data": "\nThis is some data .\n\n\nSometimes tags in the data are nested .\n\n\nIn addition to nested tags , sometimes there is also junk we need to ignore .\n\nJunk is marked by uppercase characters between asterisks and can also optionally be followed by a dash and then one or more digits . \n\nNote that *this* is just emphasized . It's not junk !\n\n",
  "entities": [
    {
      "id": 0,
      "mentions": [
        {
          "start": 9,
          "end": 18,
          "id": 0,
          "text": "some data"
        },
        {
          "start": 41,
          "end": 49,
          "id": 0,
          "text": "the data"
        }
      ]
    },
    {
      "id": 1,
      "mentions": [
        {
          "start": 33,
          "end": 60,
          "id": 1,
          "text": "tags in the data are nested"
        },
        {
          "start": 80,
          "end": 91,
          "id": 1,
          "text": "nested tags"
        }
      ]
    },
    {
      "id": 2,
      "mentions": [
        {
          "start": 118,
          "end": 122,
          "id": 2,
          "text": "junk"
        },
        {
          "start": 144,
          "end": 148,
          "id": 2,
          "text": "Junk"
        },
        {
          "start": 326,
          "end": 330,
          "id": 2,
          "text": "junk"
        }
      ]
    },
    {
      "id": 3,
      "mentions": [
        {
          "start": 1,
          "end": 5,
          "id": 3,
          "text": "This"
        }
      ]
    },
    {
      "id": 4,
      "mentions": [
        {
          "start": 289,
          "end": 295,
          "id": 4,
          "text": "*this*"
        }
      ]
    }
  ]
}

Python 3 solution here.

If you need a hint, see if you can find an event-based XML parser (or if you’re feeling really motivated, write your own).