Hello everyone!

I've recently got an idea for a Database as a Service I'd like to create. The only issue is - I don't know how to create or host a database!

I've only ever used Mongoose/mLab with Javascript, and a minimal amount of Postgres with Python.

If I'm looking to create a database that will, eventually, be able to store images, songs, and videos, where should I start my homework?

I can create the backend and the frontend with no issue - just stuck on this part here. If it's of any relevance, I most frequently use the MERN stack.

Hey Tildes, I got a job directly supervising a small team of 4 software developers. I'm very excited at the prospect and would like to put my best foot forward. To that end, I would like to have a discussion around a few topics. Feel free to expand the scope if you believe the conversation would be beneficial. I'm sure I won't be the last person to be in this position. I've done research, read, and watched videos regarding several of these questions; however, since Tilde prioritizes high-quality discussion, I thought it would be a fun opportunity to chat with others about these topics.

• As a member of a software development team, what are things that your supervisor has done that has had the greatest (a) positive and (b) negative impact?
• Supervisors, when you joined your new team, what was your methodology for reviewing the team, projects, and processes? What was the scenarios behind your review and the outcome? What would you do differently?

I'm currently looking for a new monitor and I have the ASUS MG279Q, the ASUS PG248Q and the ASUS MG278Q. Now, my setup is not the highest end, but decent with an i7 and a 1060 3GB, and there are three concerns I have currently not found an answer for:

1. The (potential) difference in quality between the MG279Q's IPS panel and the MG278's TN panel
2. And the delay difference, the TN panel having a 1ms response time and the IPS' 4ms
3. If my setup can even handle 1440p/144Hz (I don't need to play on the highest settings, nor do I need to reach those 144 FPS), in which case I would tend more towards the PG248Q

I'd love to upgrade towards 1440p as the screen real estate would be good for working (which I do a lot on the PC) and I would, I could later upgrade my GPU if the performance in games isn't satisfactory. I think my setup wouldn't have any issue handling day to day tasks and if need be I can play on lower resolutions or lower graphic settings. Also I wonder how large the difference between the IPS and TN panel is and if it's noticeable, particularly with colours.

Does anyone of you have experience with the subject or with the monitors named in particular?

Preface

In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer productivity, and the need to pay down on that debt. Today I would like to touch on a practical example that I'd encountered in a production environment.

Background

Before we can discuss the refactor itself, it's important to be on the same page regarding the technologies being used. In my case, I work with PHP utilizing a proprietary back-end framework and MongoDB as our database.

PHP is a server-side scripting language. Like many scripting languages, it's loosely typed. This has some benefits and drawbacks.

MongoDB is a document-oriented database. By default it's schema-less, allowing you to make any changes at will without an update to schema. This can blend pretty well with the loose typing of PHP. Each document is represented using a JSON-like structure and is stored in something called a "collection". For those of you accustomed to using relational database, a "collection" is analogous to a table, each document is a row, and each field in the document is a column. A typical query in the MongoDB shell would look something like this:

db.users.findOne({
    username: "Emerald_Knight"
});

The framework itself has some framework-specific objects that are held in global references. This makes them easily accessible, but naturally littering your code with a bunch of globals is both error-prone and an eyesore.

Unexpected Spaghetti

In my code base are a number of different objects that are designed to handle basic CRUD-like operations on their associated database entries. Some of these objects hold references to other objects, so naturally there is some data validation that occurs to ensure that the references are both valid and authorized. Pretty typical stuff.

What I noticed, however, is that the collection names for these database entries were littered throughout my code. This isn't necessarily a bad thing, except there were some use cases that came to mind: what if it turned out that my naming for one or more of these collections wasn't ideal? What if I wanted to change a collection name for the sake of easier management on the database end? What if I have a tendency to forget the name of a database collection and constantly have to look it up? What if I make a typo of all things? On top of that, the framework's database object was stored in a global variable.

These seemingly minor sources of technical debt end up adding up over time and could cause some serious problems in the worst case. I've had breaking bugs make their way passed QA in the past, after all.

Exchanging Spaghetti for Some Light Lasagna

The problem could be characterized simply: there were scoping problems and too many references to what were essentially magic strings. The solution, then, was to move the database object reference from global to local scope within the application code and to eliminate the problem of magic strings. Additionally, it's a good idea to avoid polluting the namespace with an over-reliance on constants, and using those constants for database calls can also become unsightly and difficult to follow as those constants could end up being generally disconnected from the objects they're associated with.

There turned out to be a nice, object-oriented, very PHP-like solution to this problem: a so-called "magic method" named "__call". This method is invoked whenever an "inaccessible" method is called on the object. Using this method, a database command executed on a non-database object could pass the command to the database object itself. If this logic were placed within an abstract class, the collection could then be specified simply as a configuration option in the inheriting class.

This is what such a solution could look like:

<?php

abstract class MyBaseObject {

    protected $db = null;
    protected $collection_name = null;

    public function __construct() {
        global $db;
        
        $this->db = $db;
    }

    public function __call($method_name, $args) {
        if(method_exists($this->db, $method_name)) {
            return $this->executeDatabaseCommand($method_name, $args);
        }

        throw new Exception(__CLASS__ . ': Method "' . $method_name . '" does not exist.');
    }

    public function executeDatabaseCommand($command, $args) {
        $collection = $this->collection_name;
        $db_collection = $this->db->$collection;

        return call_user_func_array(array($db_collection, $command), $args);
    }
}

class UserManager extends MyBaseObject {
    protected $collection_name = 'users';

    public function __construct() {
        parent::__construct();
    }
}

$user_manager = new UserManager();
$my_user = $user_manager->findOne(array('username'=>'Emerald_Knight'));

?>

This solution utilizes a single parent object which transforms a global database object reference into a local one, eliminating the scope issue. The collection name is specified as a class property of the inheriting object and only used in a single place in the parent object, eliminating the magic string and namespace polluting issues. Any time you perform queries on users, you do so by using the UserManager class, which guarantees that you will always know that your queries are being performed on the objects that you intend. And finally, if the collection name for an object class ever needs to be updated, it's a simple matter of modifying the single instance of the class property $collection_name, rather than tracking down some disconnected constant.

Limitations

This, of course, doesn't solve all of the existing problems. After all, executing the database queries for one object directly from another is still pretty bad practice, violating the principle of separation of concerns. Instead, those queries should generally be encapsulated within object methods and the objects themselves given primary responsibility in handling associated data. It's also incredibly easy to inadvertently override a database method, e.g. defining a findOne() method on UserManager, so there's still some mindfulness required on the part of the programmer.

Still, given the previous alternative, this is a pretty major improvement, especially for an initial refactor.

Final Thoughts

As always, technical debt is both necessary and inevitable. After all, in exchange for not taking the excess time and considering structuring my code this way in the beginning, I had greater initial velocity to get the project off of the ground. What's important is continually reviewing your code as you're building on top of it so that you can identify bottlenecks as they begin to strain your efficiency, and getting those bottlenecks out of the way.

In other words, even though technical debt is often necessary and is certainly inevitable, it's important to pay down on some of that debt once it starts getting expensive!

[seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly]

I've had some experience in this realm for a while now, but I'm having a little trouble with one issue in particular. Before I divulge, I'll present my thoughts on best practice and and what I've been able to implement:

• Terraform everything (in accordance to terragrunt's "style guide" i.e. organization)
  THIS IS A BIG ONE: for the jenkins master task, make sure to use the following args to make sure jenkins jobs aren't super slow as hell to start:

-Djava.awt.headless=true -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

THIS IS A GAME CHANGER (more-so on k8s clusters when the ecs plugin isn't used... hint, it's shit).

• Create an EFS (in a separate terraform module) and mount it to the jenkins ECS cluster at /var/jenkins_home. Makes jenkins much more reliable through outages and easier to upgrade.
• Run a logging agent (via docker container) like logspout or newrelic or whatever IN USER_DATA and not as a task - that way you get logs if there are issues during user_data/cloud_init... this I'm actually not sure about. Running a container outside the context of an ECS task means the ECS agent can't really track it and allocate mem/cpu properly... but it does help with user_data triage.
• Use pipelines and git plugins to drive jobs. All jenkins jobs should be in source control!
• Make sure you setup docker cleanup jobs on DAY 1! If you hace limited access to your cluster and you run out of disk due to docker cache, networks, volumes, etc... you're screwed till the admin ssh's in and runs a prune. Get a docker system prune going or the equivalent for each docker resource with appropriate filters... i.e. filter for anything older than a few days and is dangling.
• Use Jenkins Global Libraries to make Jenkinsfiles cleaner (I always just use vars instead of groovy/java style packages because it's easier and less ugly)
  Jenkinsfiles should mostly call other bash files, make files, python scripts to generate and load prop files, etc. The less logic you put in a Jenkinsfile (which is just modified groovy) the better. String interpolation, among other things, is a fuckery that we don't have time to triage.
• (out-of-scope) Move to using k8s/EKS instead of ECS asap because the ECS plugin for jenkins is absolute shit and it doesn't use priority correctly (sorry whoever developed it and... oh wait abandoned it and hasn't merged anything for years... for for real it's cool, just give admin to someone else).
• (cultural) Stop calling them slaves. "Hey @eng, we're rotating slaves due to some cache issues. If you have been affected by race conditions in that past, our new update and slave rotation should fix that. Our update may have killed your job that was running on an old slave, just wait a few and the new slaves will be ready" <--This just doesn't look good.
  Hope that was some good stuff for you guys. Maybe I'm preaching to the choir, but I've seen some pretty shit jenkins setups.

NOW FOR MY QUESTION!

Has ANYONE actually been able to setup a proper jenkins user on ECS that actually works for both a master and ephemeral jenkins-agents so that they can mount and use the docker.sock for builds without hitting permission issues? I'm talking using the ecs plugin and mounting docker.sock via that.

I have always resorted to running jenkins master and agents as root, which means you have to chmod files (super expensive time and cpu for services with tons of files). Running microservices as root is obviously bad practice, and chmod-ing a zilliion files is shit for docker cache and time... so I want to get jenkins users able to utilize the docker.sock. THIS IS SPECIFICALLY FOR THE AWS ECS AMI! I don't care about debian or old versions of docker where you could use DOCKER_OPTS. That doesn't work on the AWS Linux image.

Thanks! And happy Friday!

I have been interested in making the gradual career change to software development from my current humanities field. This stems from a handful of different places. Of course the pay and flexibility are strong drivers but I like the idea of a field that is somewhat of a creative expression; one where you can manifest your knowledge and experience into something tangible.

I have no experience with programming other than SQL use in ArcGIS and am hoping to gain some knowledge about the field; so anything would be helpful. Whether what to expect from this line of work, where someone with no experience should look to get started and what to expect, personal journeys, etc.

Cheers!

In most of my programming, I try and remain professional, and do things in a readable, maintainable way, that doesn't involve pushing the language to breaking point.

But, occasionally, I give myself free reign. What if you didn't care about the programmer who came after you? What if you didn't care about what a programmer should do in a certain circumstance?

For myself, over the years I've written and rewritten a C library, I like to call CNoEvil. Here's a little taste of what you could do:

#define EVIL_IO
#define EVIL_COROUTINE
#include "evil.h"

proc(example, int)
  static int i = 0;
  coroutine();
  While 1 then
    co_return(++i);
  end
  co_end();
  return i;
end

Main then
  displayln(example());
  displayln(example());
end

(And yes, that compiles. Without warnings. Even with -Wpedantic.)

So... Here's the challenge:

Ignoring the rules, and best practices... How can you take your favourite programming language... And make it completely unrecogniseable?

(Might be best to choose a language with macros, like Nim, Rust, any of the Lisps. Though, you can still do some impressively awful things in Java or Python, thanks to overloading inbuilt classes.)

Challenge Ideas:

• Make Python look like C
• Make Java look like Python
• Make anything look like BrainFuck

I don’t know how to really explain my fascination with programming, but I’ll try. To somebody who does it, it’s the most interesting thing in the world. It’s a game much more involved than chess, a game where you can make up your own rules and where the end result is whatever you can make of it. - Linus Torvalds

I am looking for any resource that could tell me how much knowledge and training is needed to go from a beginner to expert, in let's say application software development for example. Or in artificial intelligence.

If there isn't any one source, are there any general type of sources I can use to piece together one mega-source?

Hey y'all, first time actually posting something here! Just curious what editor people use, whether its for coding, writing, or just the occasional note, whatever. I've gone through most of the well-known ones (vim, emacs, atom, vs code for starters), but only ever really messed around with vim enough to like it, and I've also been trying out gedit for the last little while and really liking it, but I'm curious to see what other people use!

Here’s a problem I had to solve at work this week that I enjoyed solving. I think it’s a good programming challenge that will test if you really grok XML.

Your input is some XML such as this:

<DOC>
<TEXT PARTNO="000">
<TAG ID="3">This</TAG> is <TAG ID="0">some *JUNK* data</TAG> .
</TEXT>
<TEXT PARTNO="001">
*FOO* Sometimes <TAG ID="1">tags in <TAG ID="0">the data</TAG> are nested</TAG> .
</TEXT>
<TEXT PARTNO="002">
In addition to <TAG ID="1">nested tags</TAG> , sometimes there is also <TAG ID="2">junk</TAG> we need to ignore .
</TEXT>
<TEXT PARTNO="003">*BAR*-1
<TAG ID="2">Junk</TAG> is marked by uppercase characters between asterisks and can also optionally be followed by a dash and then one or more digits . *JUNK*-123
</TEXT>
<TEXT PARTNO="004">
Note that <TAG ID="4">*this*</TAG> is just emphasized . It's not <TAG ID="2">junk</TAG> !
</TEXT>
</DOC>

The above XML has so-called in-line textual annotations because the XML <TAG> elements are embedded within the document text itself.

Your goal is to convert the in-line XML annotations to so-called stand-off annotations where the text is separated from the annotations and the annotations refer to the text via slicing into the text as a character array with starting and ending character offsets. While in-line annotations are more human-readable, stand-off annotations are equally machine-readable, and stand-off annotations can be modified without changing the document content itself (the text is immutable).

The challenge, then, is to convert to a stand-off JSON format that includes the plain-text of the document and the XML tag annotations grouped by their tag element IDs. In order to preserve the annotation information from the original XML, you must keep track of each <TAG>’s starting and ending character offset within the plain-text of the document. The plain-text is defined as the character data in the XML document ignoring any junk. We’ll define junk as one or more uppercase ASCII characters [A-Z]+ between two *, and optionally a trailing dash - followed by any number of digits [0-9]+.

Here is the desired JSON output for the above example to test your solution:

{
  "data": "\nThis is some data .\n\n\nSometimes tags in the data are nested .\n\n\nIn addition to nested tags , sometimes there is also junk we need to ignore .\n\nJunk is marked by uppercase characters between asterisks and can also optionally be followed by a dash and then one or more digits . \n\nNote that *this* is just emphasized . It's not junk !\n\n",
  "entities": [
    {
      "id": 0,
      "mentions": [
        {
          "start": 9,
          "end": 18,
          "id": 0,
          "text": "some data"
        },
        {
          "start": 41,
          "end": 49,
          "id": 0,
          "text": "the data"
        }
      ]
    },
    {
      "id": 1,
      "mentions": [
        {
          "start": 33,
          "end": 60,
          "id": 1,
          "text": "tags in the data are nested"
        },
        {
          "start": 80,
          "end": 91,
          "id": 1,
          "text": "nested tags"
        }
      ]
    },
    {
      "id": 2,
      "mentions": [
        {
          "start": 118,
          "end": 122,
          "id": 2,
          "text": "junk"
        },
        {
          "start": 144,
          "end": 148,
          "id": 2,
          "text": "Junk"
        },
        {
          "start": 326,
          "end": 330,
          "id": 2,
          "text": "junk"
        }
      ]
    },
    {
      "id": 3,
      "mentions": [
        {
          "start": 1,
          "end": 5,
          "id": 3,
          "text": "This"
        }
      ]
    },
    {
      "id": 4,
      "mentions": [
        {
          "start": 289,
          "end": 295,
          "id": 4,
          "text": "*this*"
        }
      ]
    }
  ]
}

Python 3 solution here.

If you need a hint, see if you can find an event-based XML parser (or if you’re feeling really motivated, write your own).

I've been thinking a lot about dark mode lately, now that macOS and Windows 10 both officially offer some implementation of it. I think dark modes make a compelling case for eye strain prevention, but the dealbreaker for me is revealed when switching between apps and one of them isn't dark. That jarring flash of bright light completely ruins whatever gentleness the dark environment provided in the first place. So despite my curiosity I've kept everything in light mode for years, tempered by f.lux to keep myself sane after sundown.

Anyway, now that there's official OS support I'm reconsidering. I think there's a growing pro-dark movement that was just waiting for that formal recognition. Today the programs I use most all offer dark modes so I'm taking an experimental plunge. My goal: 90% elimination of white flashes while in my normal workflow.

The biggest obstacle is, not surprisingly, the web. There are some beautiful dark browser themes available but that really only affects the UI elements around the page, not the page itself. I want to darken the web too. I have a few thoughts about this:

• Plugins like this one try to automate a dark mode for every site you visit. This is hit-or-miss, resulting in ugly color combinations, sometimes unreadable text. Some methods just invert the page colors, which can lead to all sort of other visual wonkiness. I haven't found a plugin like this that isn't fiddly and annoying.
• This plugin looks interesting. From what I can tell, it uses some kind of server-side heuristics to determine the optimal way to darken every page you visit. I haven't actually tried it because I'm concerned about the privacy/security implications of sending all my web activity to this unknown third party. Or what kind of performance hit that would involve. Also, they bury this information on their site, but this is a paid service with an annual subscription.
• I'm aware of Stylish and its huge library of user-maintained custom site styles. This seemed like a good approach, except that following a recent acquisition, the new owners of Stylish betrayed their users' trust in a very shady way so I'm afraid to go near it now. If there's a credible alternative with a decent style library I'd love to know about it. Especially if there's a way to automate style application so I don't have to manually activate it for every site I visit.
• Tangentially, the W3C is having an interesting conversation about adding CSS media query support for recognizing user dark-mode preferences. This could absolutely be the future of the web(!!), but I suspect it won't because it puts the responsibility on designers to basically double the amount of work they have to do. Speaking as someone in that field, I would not want to have to add this to my already-long list of design considerations.

Are there any other good web darkening methods I've overlooked? How do you deal with the white flash problem? Should I just give up and go back to black-on-white? Interested in any and all thoughts on the matter.

It's time for another programming challenge!

Given a list of coordinate pairs on a 2D plane that describe the vertices of a polygon, determine whether the polygon is concave or convex.

Since a polygon could potentially be any shape if we don't specify which vertices connect to which, we'll assume that the coordinates are given in strict order such that adjacent coordinates in the list are connected. Specifically, if we call the list V[1, n] and say that V[i] <-> V[j] means "vertex i and vertex j are connected", then for each arbitrary V[i] we have V[i-1] <-> V[i] <-> V[i+1]. Moreover, since V[1] and V[n] are at the ends of the list, V[1] <-> V[n] holds (i.e. the list "wraps around").

Finally, for simplicity we can assume that all coordinates are unique, that all polygon descriptions generate valid polygons with 3 or more non-overlapping sides, and that, yes, we're working with coordinates that exist in the set of real numbers only. Don't over-complicate it :)

For those who want an even greater challenge, extend this out to work with 3D space!

It's been nearly a week, so it's time for another programming challenge!

This time, let's create a calculator that accepts reverse Polish notation (RPN), also known as postfix notation.

For a bit of background, RPN is where you take your two operands in an expression and place the operator after them. For example, the expression 3 + 5 would be written as 3 5 +. A more complicated expression like (5 - 3) x 8 would be written as 5 3 - 8 x, or 8 5 3 - x.

All your program has to do is accept a valid RPN string and apply the operations in the correct order to produce the expected result.