Preface

Occasionally I feel the need to touch on the subject of code quality, particularly because of the importance of its impact on technical debt, especially as I continue to encounter the effects of technical debt in my own work and do my best to manage it. It's a subject that is unfortunately not emphasized nearly enough in academia.

Background

As a refresher, technical debt is the long-term cost of the design decisions in your code. These costs can manifest in different ways, such as greater difficulty in understanding what your code is doing or making non-breaking changes to it. More generally, these costs manifest as additional time and resources being spent to make some kind of change.

Sometimes these costs aren't things you think to consider. One such consideration is how difficult it might be to upgrade a specific technology in your stack. For example, what if you've built a back-end system that integrates with AWS and you suddenly need to upgrade your SDK? In a small project this might be easy, but what if you've built a system that you've been maintaining for years and it relies heavily on AWS integrations? If the method names, namespaces, argument orders, or anything else has changed between versions, then suddenly you'll need to update every single reference to an AWS-related tool in your code to reflect those changes. In larger software projects, this could be a daunting and incredibly expensive task, spanning potentially weeks or even months of work and testing.

That is, unless you keep those references to a minimum.

A Toy Example

This is where "wrapping" your external libraries comes into play. The concept of "wrapping" basically means to create some other function or object that takes care of operating the functions or object methods that you really want to target. One example might look like this:

<?php

class ImportedClass {
    public function methodThatMightBecomeModified($arg1, $arg2) {
        // Do something.
    }
}

class ImportedClassWrapper {
    private $class_instance = null;

    private function getInstance() {
        if(is_null($this->class_instance)) {
            $this->class_instance = new ImportedClass();
        }

        return $this->class_instance;
    }

    public function wrappedMethod($arg1, $arg2) {
        return $this->getInstance()->methodThatMightBecomeModified($arg1, $arg2);
    }
}

?>

Updating Tools Doesn't Have to Suck

Imagine that our ImportedClass has some important new features that we need to make use of that are only available in the most recent version, and we're several versions behind. The problem, of course, is that there were a lot of changes that ended up being made between our current version and the new version. For example, ImportedClass is now called NewImportedClass. On top of that, methodThatMightBecomeModified is now called methodThatWasModified, and the argument order ended up getting switched around!

Now imagine that we were directly calling new ImportedClass() in many different places in our code, as well as directly invoking methodThatMightBecomeModified:

<?php

$imported_class_instance = new ImportedClass();
$imported_class_instance->methodThatMightBeModified($val1, $val2);

?>

For every single instance in our code, we need to perform a replacement. There is a linear or--in terms of Big-O notation--a complexity of O(n) to make these replacements. If we assume that we only ever used this one method, and we used it 100 times, then there are 100 instances of new ImportClass() to update and another 100 instances of the method invocation, equaling 200 lines of code to change. Furthermore, we need to remember each of the replacements that need to be made and carefully avoid making any errors in the process. This is clearly non-ideal.

Now imagine that we chose instead to use the wrapper object:

<?php

$imported_class_wrapper = new ImportedClassWrapper();
$imported_class_wrapper->wrappedMethod($val1, $val2);

?>

Our updates are now limited only to the wrapper class:

<?php

class ImportedClassWrapper {
    private $class_instance = null;

    private function getInstance() {
        if(is_null($this->class_instance)) {
            $this->class_instance = new NewImportedClass();
        }

        return $this->class_instance;
    }

    public function wrappedMethod($arg1, $arg2) {
        return $this->getInstance()->methodThatWasModified($arg2, $arg1);
    }
}

?>

Rather than making changes to 200 lines of code, we've now made changes to only 2. What was once an O(n) complexity change has now turned into an O(1) complexity change to make this upgrade. Not bad for a few extra lines of code!

A Practical Example

Toy problems are all well and good, but how does this translate to reality?

Well, I ran into such a problem myself once. Running MongoDB with PHP requires the use of an external driver, and this driver provides an object representing a MongoDB ObjectId. I needed to perform a migration from one hosting provider over to a new cloud hosting provider, with the application and database services, which were originally hosted on the same physical machine, hosted on separate servers. For security reasons, this required an upgrade to a newer version of MongoDB, which in turn required an upgrade to a newer version of the driver.

This upgrade resulted in many of the calls to new MongoId() failing, because the old version of the driver would accept empty strings and other invalid ID strings and default to generating a new ObjectId, whereas the new version of the driver treated invalid ID strings as failing errors. And there were many, many cases where invalid strings were being passed into the constructor.

Even after spending hours replacing the (literally) several dozen instances of the constructor calls, there were still some places in the code where invalid strings managed to get passed in. This made for a very costly upgrade.

The bugs were easy to fix after the initial replacements, though. After wrapping new MongoId() inside of a wrapper function, a few additional conditional statements inside of the new function resolved the bugs without having to dig around the rest of the code base.

Final Thoughts

This is one of those lessons that you don't fully appreciate until you've experienced the technical debt of an unwrapped external library first-hand. Code quality is an active effort, but a worthwhile one. It requires you to be willing to throw away potentially hours or even days of work when you realize that something needs to change, because you're thinking about how to keep yourself from banging your head against a wall later down the line instead of thinking only about how to finish up your current task.

"Work smarter, not harder" means putting in some hard work upfront to keep your technical debt under control.

That's all for now, and remember: don't be fools, wrap your external tools.

The programming challenges have kind of come to a grinding halt recently. I think it's time to get another challenge started!

Given a grid of symbols, representing a simple binary state of "filled" or "unfilled", determine whether or not a square is present on the grid. Squares must be 2x2 in size or larger, must be completely solid (i.e. all symbols in the NxN space are "filled"), and must not be directly adjacent to any other filled spaces.

Example, where 0 is "empty" and 1 is "filled":

000000
011100
011100
011100
000010

// Returns true.

000000
011100
011100
011110
000000

// Returns false.

000000
011100
010100
011100
000000

// Returns false.

For those who want a greater challenge, try any of the following:

1. Get a count of all squares.
2. Detect squares that are touching (but not as a rectangle).
3. Detect other specific shapes like triangles or circles (you will need to be creative).
4. If doing (1) and (3), count shapes separately based on type.
5. Detect shapes within unfilled space as well (a checkerboard pattern is a great use case).

Hello everyone!

I've recently got an idea for a Database as a Service I'd like to create. The only issue is - I don't know how to create or host a database!

I've only ever used Mongoose/mLab with Javascript, and a minimal amount of Postgres with Python.

If I'm looking to create a database that will, eventually, be able to store images, songs, and videos, where should I start my homework?

I can create the backend and the frontend with no issue - just stuck on this part here. If it's of any relevance, I most frequently use the MERN stack.

Hey Tildes, I got a job directly supervising a small team of 4 software developers. I'm very excited at the prospect and would like to put my best foot forward. To that end, I would like to have a discussion around a few topics. Feel free to expand the scope if you believe the conversation would be beneficial. I'm sure I won't be the last person to be in this position. I've done research, read, and watched videos regarding several of these questions; however, since Tilde prioritizes high-quality discussion, I thought it would be a fun opportunity to chat with others about these topics.

• As a member of a software development team, what are things that your supervisor has done that has had the greatest (a) positive and (b) negative impact?
• Supervisors, when you joined your new team, what was your methodology for reviewing the team, projects, and processes? What was the scenarios behind your review and the outcome? What would you do differently?

I'm currently looking for a new monitor and I have the ASUS MG279Q, the ASUS PG248Q and the ASUS MG278Q. Now, my setup is not the highest end, but decent with an i7 and a 1060 3GB, and there are three concerns I have currently not found an answer for:

1. The (potential) difference in quality between the MG279Q's IPS panel and the MG278's TN panel
2. And the delay difference, the TN panel having a 1ms response time and the IPS' 4ms
3. If my setup can even handle 1440p/144Hz (I don't need to play on the highest settings, nor do I need to reach those 144 FPS), in which case I would tend more towards the PG248Q

I'd love to upgrade towards 1440p as the screen real estate would be good for working (which I do a lot on the PC) and I would, I could later upgrade my GPU if the performance in games isn't satisfactory. I think my setup wouldn't have any issue handling day to day tasks and if need be I can play on lower resolutions or lower graphic settings. Also I wonder how large the difference between the IPS and TN panel is and if it's noticeable, particularly with colours.

Does anyone of you have experience with the subject or with the monitors named in particular?

Preface

In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer productivity, and the need to pay down on that debt. Today I would like to touch on a practical example that I'd encountered in a production environment.

Background

Before we can discuss the refactor itself, it's important to be on the same page regarding the technologies being used. In my case, I work with PHP utilizing a proprietary back-end framework and MongoDB as our database.

PHP is a server-side scripting language. Like many scripting languages, it's loosely typed. This has some benefits and drawbacks.

MongoDB is a document-oriented database. By default it's schema-less, allowing you to make any changes at will without an update to schema. This can blend pretty well with the loose typing of PHP. Each document is represented using a JSON-like structure and is stored in something called a "collection". For those of you accustomed to using relational database, a "collection" is analogous to a table, each document is a row, and each field in the document is a column. A typical query in the MongoDB shell would look something like this:

db.users.findOne({
    username: "Emerald_Knight"
});

The framework itself has some framework-specific objects that are held in global references. This makes them easily accessible, but naturally littering your code with a bunch of globals is both error-prone and an eyesore.

Unexpected Spaghetti

In my code base are a number of different objects that are designed to handle basic CRUD-like operations on their associated database entries. Some of these objects hold references to other objects, so naturally there is some data validation that occurs to ensure that the references are both valid and authorized. Pretty typical stuff.

What I noticed, however, is that the collection names for these database entries were littered throughout my code. This isn't necessarily a bad thing, except there were some use cases that came to mind: what if it turned out that my naming for one or more of these collections wasn't ideal? What if I wanted to change a collection name for the sake of easier management on the database end? What if I have a tendency to forget the name of a database collection and constantly have to look it up? What if I make a typo of all things? On top of that, the framework's database object was stored in a global variable.

These seemingly minor sources of technical debt end up adding up over time and could cause some serious problems in the worst case. I've had breaking bugs make their way passed QA in the past, after all.

Exchanging Spaghetti for Some Light Lasagna

The problem could be characterized simply: there were scoping problems and too many references to what were essentially magic strings. The solution, then, was to move the database object reference from global to local scope within the application code and to eliminate the problem of magic strings. Additionally, it's a good idea to avoid polluting the namespace with an over-reliance on constants, and using those constants for database calls can also become unsightly and difficult to follow as those constants could end up being generally disconnected from the objects they're associated with.

There turned out to be a nice, object-oriented, very PHP-like solution to this problem: a so-called "magic method" named "__call". This method is invoked whenever an "inaccessible" method is called on the object. Using this method, a database command executed on a non-database object could pass the command to the database object itself. If this logic were placed within an abstract class, the collection could then be specified simply as a configuration option in the inheriting class.

This is what such a solution could look like:

<?php

abstract class MyBaseObject {

    protected $db = null;
    protected $collection_name = null;

    public function __construct() {
        global $db;
        
        $this->db = $db;
    }

    public function __call($method_name, $args) {
        if(method_exists($this->db, $method_name)) {
            return $this->executeDatabaseCommand($method_name, $args);
        }

        throw new Exception(__CLASS__ . ': Method "' . $method_name . '" does not exist.');
    }

    public function executeDatabaseCommand($command, $args) {
        $collection = $this->collection_name;
        $db_collection = $this->db->$collection;

        return call_user_func_array(array($db_collection, $command), $args);
    }
}

class UserManager extends MyBaseObject {
    protected $collection_name = 'users';

    public function __construct() {
        parent::__construct();
    }
}

$user_manager = new UserManager();
$my_user = $user_manager->findOne(array('username'=>'Emerald_Knight'));

?>

This solution utilizes a single parent object which transforms a global database object reference into a local one, eliminating the scope issue. The collection name is specified as a class property of the inheriting object and only used in a single place in the parent object, eliminating the magic string and namespace polluting issues. Any time you perform queries on users, you do so by using the UserManager class, which guarantees that you will always know that your queries are being performed on the objects that you intend. And finally, if the collection name for an object class ever needs to be updated, it's a simple matter of modifying the single instance of the class property $collection_name, rather than tracking down some disconnected constant.

Limitations

This, of course, doesn't solve all of the existing problems. After all, executing the database queries for one object directly from another is still pretty bad practice, violating the principle of separation of concerns. Instead, those queries should generally be encapsulated within object methods and the objects themselves given primary responsibility in handling associated data. It's also incredibly easy to inadvertently override a database method, e.g. defining a findOne() method on UserManager, so there's still some mindfulness required on the part of the programmer.

Still, given the previous alternative, this is a pretty major improvement, especially for an initial refactor.

Final Thoughts

As always, technical debt is both necessary and inevitable. After all, in exchange for not taking the excess time and considering structuring my code this way in the beginning, I had greater initial velocity to get the project off of the ground. What's important is continually reviewing your code as you're building on top of it so that you can identify bottlenecks as they begin to strain your efficiency, and getting those bottlenecks out of the way.

In other words, even though technical debt is often necessary and is certainly inevitable, it's important to pay down on some of that debt once it starts getting expensive!

[seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly]

I've had some experience in this realm for a while now, but I'm having a little trouble with one issue in particular. Before I divulge, I'll present my thoughts on best practice and and what I've been able to implement:

• Terraform everything (in accordance to terragrunt's "style guide" i.e. organization)
  THIS IS A BIG ONE: for the jenkins master task, make sure to use the following args to make sure jenkins jobs aren't super slow as hell to start:

-Djava.awt.headless=true -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

THIS IS A GAME CHANGER (more-so on k8s clusters when the ecs plugin isn't used... hint, it's shit).

• Create an EFS (in a separate terraform module) and mount it to the jenkins ECS cluster at /var/jenkins_home. Makes jenkins much more reliable through outages and easier to upgrade.
• Run a logging agent (via docker container) like logspout or newrelic or whatever IN USER_DATA and not as a task - that way you get logs if there are issues during user_data/cloud_init... this I'm actually not sure about. Running a container outside the context of an ECS task means the ECS agent can't really track it and allocate mem/cpu properly... but it does help with user_data triage.
• Use pipelines and git plugins to drive jobs. All jenkins jobs should be in source control!
• Make sure you setup docker cleanup jobs on DAY 1! If you hace limited access to your cluster and you run out of disk due to docker cache, networks, volumes, etc... you're screwed till the admin ssh's in and runs a prune. Get a docker system prune going or the equivalent for each docker resource with appropriate filters... i.e. filter for anything older than a few days and is dangling.
• Use Jenkins Global Libraries to make Jenkinsfiles cleaner (I always just use vars instead of groovy/java style packages because it's easier and less ugly)
  Jenkinsfiles should mostly call other bash files, make files, python scripts to generate and load prop files, etc. The less logic you put in a Jenkinsfile (which is just modified groovy) the better. String interpolation, among other things, is a fuckery that we don't have time to triage.
• (out-of-scope) Move to using k8s/EKS instead of ECS asap because the ECS plugin for jenkins is absolute shit and it doesn't use priority correctly (sorry whoever developed it and... oh wait abandoned it and hasn't merged anything for years... for for real it's cool, just give admin to someone else).
• (cultural) Stop calling them slaves. "Hey @eng, we're rotating slaves due to some cache issues. If you have been affected by race conditions in that past, our new update and slave rotation should fix that. Our update may have killed your job that was running on an old slave, just wait a few and the new slaves will be ready" <--This just doesn't look good.
  Hope that was some good stuff for you guys. Maybe I'm preaching to the choir, but I've seen some pretty shit jenkins setups.

NOW FOR MY QUESTION!

Has ANYONE actually been able to setup a proper jenkins user on ECS that actually works for both a master and ephemeral jenkins-agents so that they can mount and use the docker.sock for builds without hitting permission issues? I'm talking using the ecs plugin and mounting docker.sock via that.

I have always resorted to running jenkins master and agents as root, which means you have to chmod files (super expensive time and cpu for services with tons of files). Running microservices as root is obviously bad practice, and chmod-ing a zilliion files is shit for docker cache and time... so I want to get jenkins users able to utilize the docker.sock. THIS IS SPECIFICALLY FOR THE AWS ECS AMI! I don't care about debian or old versions of docker where you could use DOCKER_OPTS. That doesn't work on the AWS Linux image.

Thanks! And happy Friday!

I have been interested in making the gradual career change to software development from my current humanities field. This stems from a handful of different places. Of course the pay and flexibility are strong drivers but I like the idea of a field that is somewhat of a creative expression; one where you can manifest your knowledge and experience into something tangible.

I have no experience with programming other than SQL use in ArcGIS and am hoping to gain some knowledge about the field; so anything would be helpful. Whether what to expect from this line of work, where someone with no experience should look to get started and what to expect, personal journeys, etc.

Cheers!

In most of my programming, I try and remain professional, and do things in a readable, maintainable way, that doesn't involve pushing the language to breaking point.

But, occasionally, I give myself free reign. What if you didn't care about the programmer who came after you? What if you didn't care about what a programmer should do in a certain circumstance?

For myself, over the years I've written and rewritten a C library, I like to call CNoEvil. Here's a little taste of what you could do:

#define EVIL_IO
#define EVIL_COROUTINE
#include "evil.h"

proc(example, int)
  static int i = 0;
  coroutine();
  While 1 then
    co_return(++i);
  end
  co_end();
  return i;
end

Main then
  displayln(example());
  displayln(example());
end

(And yes, that compiles. Without warnings. Even with -Wpedantic.)

So... Here's the challenge:

Ignoring the rules, and best practices... How can you take your favourite programming language... And make it completely unrecogniseable?

(Might be best to choose a language with macros, like Nim, Rust, any of the Lisps. Though, you can still do some impressively awful things in Java or Python, thanks to overloading inbuilt classes.)

Challenge Ideas:

• Make Python look like C
• Make Java look like Python
• Make anything look like BrainFuck

I don’t know how to really explain my fascination with programming, but I’ll try. To somebody who does it, it’s the most interesting thing in the world. It’s a game much more involved than chess, a game where you can make up your own rules and where the end result is whatever you can make of it. - Linus Torvalds

I am looking for any resource that could tell me how much knowledge and training is needed to go from a beginner to expert, in let's say application software development for example. Or in artificial intelligence.

If there isn't any one source, are there any general type of sources I can use to piece together one mega-source?