Preface

It's not uncommon for a written piece of code to be both brief and functionality correct, yet difficult to reason about. This is especially true of recursive algorithms, which can require some amount of simulating the algorithm mentally (or on a whiteboard) on smaller problems to try to understand the underlying logic. The more you have to perform these manual simulations, the more difficult it becomes to track what exactly is going on at any stage of computation. It's also not uncommon that these algorithms can be made easier to reason about with relatively small changes, particularly in the way you conceptualize the solution to the problem. Our goal will be to take a brief tour into what these changes might look like and why they are effective at reducing our mental overhead.

Background

We will consider the case of the subset sum problem, which is essentially a special case of the knapsack problem where you have a finite number of each item and each item's value is equal to its weight. In short, the problem is summarized as one of the following:

• Given a set of numbers, is there a subset whose sum is exactly equal to some target value?

• Given a set of numbers, what is the subset whose sum is the closest to some target value without exceeding it?

For example, given the set of numbers {1, 3, 3, 5} and a target value of 9, the answer for both of those questions is {1, 3, 5} because the sum of those numbers is 9. For a target value of 10, however, the first question has no solution because no combination of numbers in the set {1, 3, 3, 5} produces a total of 10, but the second question produces a solution of {1, 3, 5} because 9 is the closest value to 10 that those numbers can produce without going over.

A Greedy Example

We'll stick to the much simpler case of finding an exact match to our target value so we don't have to track what the highest value found so far is. To make things even simpler, we'll consider the case where all numbers are positive, non-zero integers. This problem can be solved with some naive recursion--simply try all combinations until either a solution is found or all combinations have been exhausted. While more efficient solutions exist, naive recursion is the easiest to conceptualize.

An initial assessment of the problem seems simple enough. Our solution is defined as the set of array elements whose total is equal to our target value. To achieve this, we loop through each of the elements in the array, try combinations with all of the remaining elements, and keep track of what the current total is so we can compare it to our target. If we find an exact match, we return an array containing the matching elements, otherwise we return nothing. This gives us something like the following:

function subsetSum($target_sum, $values, $total = 0) {
    // Base case: a total exceeding our target sum is a failure.
    if($total > $target_sum) {
        return null;
    }

    // Base case: a total matching our target sum means we've found a match.
    if($total == $target_sum) {
        return array();
    }

    foreach($values as $index=>$value) {
        // Recursive case: try combining the current array element with the remaining elements.
        $result = subsetSum($target_sum, array_slice($values, $index + 1), $total + $value);

        if(!is_null($result)) {
            return array_merge(array($value), $result);
        }
    }

    return null;
}

Your Scope is Leaking

This solution works. It's functionally correct and will produce a valid result every single time. From a purely functional perspective, nothing is wrong with it at all; however, it's not easy to follow what's going on despite how short the code is. If we look closely, we can tell that there are a few major problems:

• It's not obvious at first glance whether or not the programmer is expected to provide the third argument. While a default value is provided, it's not clear if this value is only a default that should be overridden or if the value should be left untouched. This ambiguity means relying on documentation to explain the intention of the third argument, which may still be ignored by an inattentive developer.

• The base case where a failure occurs, i.e. when the accumulated total exceeds the target sum, occurs one stack frame further into the recursion than when the total has been incremented. This forces us to consider not only the current iteration of recursion, but one additional iteration deeper in order to track the flow of execution. Ideally an iteration of recursion should be conceptually isolated from any other, limiting our mental scope to only the current iteration.

• We're propagating an accumulating total that starts from 0 and increments toward our target value, forcing us to to track two different values simultaneously. Ideally we would only track one value if possible. If we can manage that, then the ambiguity of the third argument will be eliminated along with the argument itself.

Overall, the amount of code that the programmer needs to look at and the amount of branching they need to follow manually is excessive. The function is only 22 lines long, including whitespace and comments, and yet the amount of effort it takes to ensure you're understanding the flow of execution correctly is pretty significant. This is a pretty good indicator that we probably did something wrong. Something so simple and short shouldn't take so much effort to understand.

Patching the Leak

Now that we've assessed the problems, we can see that our original solution isn't going to cut it. We have a couple of ways we could approach fixing our function: we can either attempt to translate the abstract problems into tangible solutions or we can modify the way we've conceptualized the solution. With that in mind, let's take a second crack at this problem by trying the latter.

We've tried taking a look at this problem from a top-down perspective: "given a target value, are there any elements that produce a sum exactly equal to it?" Clearly this perspective failed us. Instead, let's try flipping the equation: "given an array element, can it be summed with others to produce the target value?"

This fundamentally changes the way we can think about the problem. Previously we were hung up on the idea of keeping track of the current total sum of the elements we've encountered so far, but that approach is incompatible with the way we're thinking of this problem now. Rather than incrementing a total, we now find ourselves having to do something entirely different: if we want to know if a given array element is part of the solution, we need to first subtract the element from the problem and find out if the smaller problem has a solution. That is, to find if the element 3 is part of the solution for the target sum of 8, then we're really asking if 3 + solutionFor(5) is valid.

The new solution therefore involves looping over our array elements just as before, but this time we check if there is a solution for the target sum minus the current array element:

function subsetSum($target_sum, $values) {
    // Base case: the solution to the target sum of 0 is the empty set.
    if($target_sum === 0) {
        return array();
    }

    foreach($values as $index=>$value) {
        // Base case: any element larger than our target sum cannot be part of the solution.
        if($value > $target_sum) {
            continue;
        }

        // Recursive case: do the remaining elements create a solution for the sub-problem?
        $result = subsetSum($target_sum - $value, array_slice($values, $index + 1));

        if(!is_null($result)) {
            return array_merge(array($value), $result);
        }
    }

    return null;
}

A Brief Review

With the changes now in place, let's compare our two functions and, more importantly, compare our new function to the problems we assessed with the original. A few brief points:

• Both functions are the same exact length, being only 22 lines long with the same number of comments and an identical amount of whitespace.

• Both functions touch the same number of elements and produce the same output given the same input. Apart from a change in execution order of a base case, functionality is nearly identical.

• The new function no longer requires thinking about the scope of next iteration of recursion to determine whether or not an array element is included in the result set. The base case for exceeding the target sum now occurs prior to recursion, keeping the scope of the value comparison nearest where those values are defined.

• The new function no longer uses a third accumulator argument, reducing the number of values to be tracked and removing the issue of ambiguity with whether or not to include the third argument in top-level calls.

• The new function is now defined in terms of finding the solutions to increasingly smaller target sums, making it easier to determine functional correctness.

Considering all of the above, we can confidently state that the second function is easier to follow, easier to verify functional correctness for, and less confusing for anyone who needs to use it. Although the two functions are nearly identical, the second version is clearly and objectively better than the original. This is because despite both being functionally correct, the first function does a poor job at accurately defining the problem it's solving while the second function is clear and accurate in its definition.

Correct code isn't necessarily accurate code. Anyone can write code that works, but writing code that accurately defines a problem can mean the difference between understanding what you're looking at, and being completely bewildered at how, or even why, your code works in the first place.

Final Thoughts

Accurately defining a problem in code isn't easy. Sometimes you'll get it right, but more often than not you'll get it wrong on the first go, and it's only after you've had some distance from you original solution that you realize that you should've done things differently. Despite that, understanding the difference between functional correctness and accuracy gives you the opportunity to watch for obvious inaccuracies and keep them to a minimum.

In the end, even functionally correct, inaccurate code is worth more than no code at all. No amount of theory is a replacement for practical experience. The only way to get better is to mess up, assess why you messed up, and make things just a little bit better the next time around. Theory just makes that a little easier.

Preface

Conceptual models in programming are essential for being able to reason about problems. We see this through code all the time, with implementation details hidden away behind abstractions like functions and objects so that we can ignore the cumbersome details and focus only on the details that matter. Without these abstractions and conceptual models, we might find ourselves overwhelmed by the size and complexity of the problem we’re facing. Of these conceptual models, one of the most easily neglected is that of data and object structure.

Data Types Galore

Possibly one of the most overwhelming aspects of conceptualizing data and object structure is the sheer breadth of data types available. Depending on the programming language you’re working with, you may find that you have more than several dozens of object classes already defined as part of the language’s core; primitives like booleans, ints, unsigned ints, floats, doubles, longs, strings, chars, and possibly others; arrays that can contain any of the objects or primitives, and even other arrays; and several other data structures like queues, vectors, and mixed-type collections, among others.

With so many types of data, it’s incredibly easy to lose track in a sea of type declarations and find yourself confused and unsure of where to go.

Tree’s Company

Let’s start by trying to make these data types a little less overwhelming. Rather than thinking strictly of types, let’s classify them. We can group all data types into one of three basic classifications:

1. Objects, which contain key/value pairs. For example, an object property that stores a string.
2. Arrays, which contain some arbitrary number of values.
3. Primitives, which contain nothing. They’re simply a “flat” data value.

We can also make a couple of additional notes. First, arrays and objects are very similar; both contain references to internal data, but the way that data is referenced differs. In particular, objects have named keys while arrays have numeric, zero-indexed keys. In a sense, arrays are a special case of objects where the keys are more strictly typed. From this, we can condense the classifications of objects and arrays into the more general “container” classification.

With that in mind, we now have the following classifications:

1. Containers.
2. Primitives.

We can now generally state that containers may contain other containers and primitives, and primitives may not contain anything. In other words, all data structures are a composition of containers and/or primitives, where containers may accept containers and/or primitives and primitives may not accept anything. More experienced programmers should notice something very familiar about this description--we’re basically describing a tree structure! Primitive types and empty containers act as the leaves in a tree, whereas objects and arrays act as the nodes.

Trees Help You Breathe

Okay, great. So what’s the big deal, anyway? We’ve now traded a bunch of concrete data types that we can actually think about and abstracted them away into this nebulous mess of containers and primitives. What do we get out of this?

A common mistake many programmers make is planning their data types out from the very beginning. Rather than planning out an abstraction for their data and object architecture, it’s easy to immediately find yourself focusing too much on the concrete implementation details.

Imagine, for example, modeling a user account for an online payment system. A common feature to include is the ability to store payment information for auto-pay, and payment methods typically take the form of some combination of credit/debit cards and bank accounts. If we focus on implementation details from the beginning, then we may find ourselves with something like this in a first iteration:

UserAccount: {
    username: String,
    password: String,
    payment_methods: PaymentMethod[]
}

PaymentMethod: {
    account_name: String,
    account_type: Enum,
    account_holder: String,
    number: String,
    routing_number: String?,
    cvv: String?,
    expiration_date: DateString?
}

We then find ourselves realizing that PaymentMethod is an unnecessary mess of optional values and needing to refactor it. Odds are we would break it off immediately into separate account types and make a note that they both implement some interface. We may also find that, as a result, remodeling the PaymentMethod could result in the need to remodel the UserAccount. For more deeply nested data structures, a single change deeper within the structure could result in those changes cascading all the way to the top-level object. If we have multiple objects, then these changes could propagate to them as well. And what if we decide a type needs to be changed, like deciding that our expiration date needs to be some sort of date object? Or what if we decide that we want to modify our property names? We’re then stuck having to update these definitions as we go along. What if we decide that we don't want an interface for different payment method types after all and instead want separate collections for each type? Then including the interface consideration will have proven to be a waste of time. The end result is that before we’ve even touched a single line of code, we’ve already found ourselves stuck with a bunch of technical debt, and we’re only in our initial planning stages!

To alleviate these kinds of problems, it’s far better to just ignore the implementation details. By doing so, we may find ourselves with something like this:

UserAccount: {
    Username,
    Password,
    PaymentMethods
}

PaymentMethods: // TODO: Decide on this container’s structure.

CardAccount: {
    AccountName,
    CardHolder,
    CardNumber,
    CVV,
    ExpirationDate,
    CardType
}

BankAccount: {
    AccountName,
    AccountNumber,
    RoutingNumber,
    AccountType
}

A few important notes about what we’ve just done here:

1. We don’t specify any concrete data types.
2. All fields within our models have the capacity to be either containers or primitives.
3. We’re able to defer a model’s structural definition without affecting the pace of our planning.
4. Any changes to a particular field type will automatically propagate in our structural definitions, making it trivial to create a definition like ExpirationDate: String and later change it to ExpirationDate: DateObject.
5. The amount of information we need to think about is reduced down to the very bare minimum.
6. By deferring the definition of the PaymentMethods structure, we find ourselves more inclined to focus on the more concrete payment method definitions from the very beginning, rather than trying to force them to be compatible through an interface.
7. We focused only on data representation, ensuring that representation and implementation are both separate and can be handled differently if needed.

SOLIDifying Our Conceptual Model

In object-oriented programming (OOP), there’s a generally recommended set of principles to follow, represented by the acronym “SOLID”:

• Single responsibility.
• Open/closed.
• Liskov substitution.
• Interface segregation.
• Dependency inversion.

These “SOLID” principles were defined to help resolve common, recurring design problems and anti-patterns in OOP.

Of particular note for us is the last one, the “dependency inversion” principle. The idea behind this principle is that implementation details should depend on abstractions, not the other way around. Our new conceptual model obeys the dependency inversion principle by prioritizing a focus on abstractions while leaving implementation details to the future class definitions that are based on our abstractions. By doing so, we limit the elements involved in our planning and problem-solving stages to only what is necessary.

Final Thoughts

The consequences of such a conceptual model extend well beyond simply planning out data and object structures. For example, if implemented as an actual programming or language construct, you could make the parsing of your data fairly simple. By implementing an object parser that performs reflection on some passed object, you can extract all of the publicly accessible object properties of the target object and the data contained therein. Thus, if your language doesn’t have a built-in JSON encoding function and no library yet exists, you could recursively traverse your data structure to generate the appropriate JSON with very little effort.

Many of the most fundamental programming concepts, like data structures ultimately being nothing more than trees at their most abstract representation, are things we tend to take for granted and think very little about. By making ourselves conscious of these fundamental concepts, however, we can more effectively take advantage of them.

Additionally, successful programmers typically solve a programming problem before they’ve ever written a single line of code. Whether or not they’re conscious of it, the tools they use to solve these problems effectively consist largely of the myriad conceptual models they’ve collected and developed over time, and the experience they’ve accumulated to determine which conceptual models need to be utilized to solve a particular problem.

Even when you have a solid grasp of your programming fundamentals, you should always revisit them every now and then. Sometimes there are details that you may have missed or just couldn’t fully appreciate when you learned about them. This is something that I’m continually reminded of as I continue on in my own career growth, and I hope that I can continue passing these lessons on to others.

As always, I'm absolutely open to feedback and questions!

Preface

Recently I briefly touched on the subject of cyclomatic complexity. This is an important concept for any programmer to understand and think about as they write their code. In order to provide a more solid understanding of the subject, however, I feel that I need to address the topic more thoroughly with a more practical example.

What is cyclomatic complexity?

The concept of "cyclomatic complexity" is simple: the more conditional branching and looping in your code, the more complex--and therefore the more difficult to maintain--that code is. We can visualize this complexity by drawing a diagram that illustrates the flow of logic in our program. For example, let's take the following toy example of a user login attempt:

<?php

$login_data = getLoginCredentialsFromInput();

$login_succeeded = false;
$error = '';
if(usernameExists($login_data['username'])) {
    $user = getUser($login_data['username']);
    
    if(!isDeleted($user)) {
        if(!isBanned($user)) {
            if(!loginRateLimitReached($user)) {
                if(passwordMatches($user, $login_data['password'])) {
                    loginUser($user);
                    $login_succeeded = true;
                } else {
                    $error = getBadPasswordError();
                    logBadLoginAttempt();
                }
            } else {
                $error = getLoginRateLimitError($user);
            }
        } else {
            $error = getUserBannedError($user);
        }
    } else {
        $error = getUserDeletedError($user);
    }
} else {
    $error = getBadUsernameError($login_data['username']);
}

if($login_succeeded) {
    sendSuccessResponse();
} else {
    sendErrorResponse($error);
}

?>

A diagram for this logic might look something like this:

+-----------------+
|                 |
|  Program Start  |
|                 |
+--------+--------+
         |
         |
         v
+--------+--------+    +-----------------+
|                 |    |                 |
|    Username     +--->+    Set Error    +--+
|    Exists?      | No |                 |  |
|                 |    +-----------------+  |
+--------+--------+                         |
         |                                  |
     Yes |                                  |
         v                                  |
+--------+--------+    +-----------------+  |
|                 |    |                 |  |
|  User Deleted?  +--->+    Set Error    +->+
|                 | Yes|                 |  |
+--------+--------+    +-----------------+  |
         |                                  |
      No |                                  |
         v                                  |
+--------+--------+    +-----------------+  |
|                 |    |                 |  |
|  User Banned?   +--->+    Set Error    +->+
|                 | Yes|                 |  |
+--------+--------+    +-----------------+  |
         |                                  |
      No |                                  |
         v                                  |
+--------+--------+    +-----------------+  |
|                 |    |                 |  |
|   Login Rate    +--->+    Set Error    +->+
| Limit Reached?  | Yes|                 |  |
|                 |    +-----------------+  |
+--------+--------+                         |
         |                                  |
      No |                                  |
         v                                  |
+--------+--------+    +-----------------+  |
|                 |    |                 |  |
|Password Matches?+--->+    Set Error    +->+
|                 | No |                 |  |
+--------+--------+    +-----------------+  |
         |                                  |
     Yes |                                  |
         v                                  |
+--------+--------+    +----------+         |
|                 |    |          |         |
|   Login User    +--->+ Converge +<--------+
|                 |    |          |
+-----------------+    +---+------+
                           |
                           |
         +-----------------+
         |
         v
+--------+--------+
|                 |
|   Succeeded?    +-------------+
|                 | No          |
+--------+--------+             |
         |                      |
     Yes |                      |
         v                      v
+--------+--------+    +--------+--------+
|                 |    |                 |
|  Send Success   |    |   Send Error    |
|    Message      |    |    Message      |
|                 |    |                 |
+-----------------+    +-----------------+

It's important to note that between nodes in this directed graph, you can find certain enclosed regions being formed. Specifically, each conditional branch that converges back into the main line of execution generates an additional region. The number of these distinct enclosed regions is directly proportional to the level of cyclomatic complexity of the system--that is, more regions means more complicated code.

Clocking out early.

There's an important piece of information I noted when describing the above example:

. . . each conditional branch that converges back into the main line of execution generates an additional region.

The above example is made complex largely due to an attempt to create a single exit point at the end of the program logic, causing these conditional branches to converge and thus generate the additional enclosed regions within our diagram.

But what if we stopped trying to converge back into the main line of execution? What if, instead, we decided to interrupt the program execution as soon as we encountered an error? Our code might look something like this:

<?php

$login_data = getLoginCredentialsFromInput();

if(!usernameExists($login_data['username'])) {
    sendErrorResponse(getBadUsernameError($login_data['username']));
    return;
}

$user = getUser($login_data['username']);
if(isDeleted($user)) {
    sendErrorResponse(getUserDeletedError($user));
    return;
}

if(isBanned($user)) {
    sendErrorResponse(getUserBannedError($user));
    return;
}

if(loginRateLimitReached($user)) {
    logBadLoginAttempt($user);
    sendErrorResponse(getLoginRateLimitError($user));
    return;
}

if(!passwordMatches($user, $login_data['password'])) {
    logBadLoginAttempt($user);
    sendErrorResponse(getBadPasswordError());
    return;
}

loginUser($user);
sendSuccessResponse();

?>

Before we've even constructed a diagram for this logic, we can already see just how much simpler this logic is. We don't need to traverse a tree of if statements to determine which error message has priority to be sent out, we don't need to attempt to follow indentation levels, and our behavior on success is right at the very end and at the lowest level of indentation, where it's easily and obviously located at a glance.

Now, however, let's verify this reduction in complexity by examining the associated diagram:

+-----------------+
|                 |
|  Program Start  |
|                 |
+--------+--------+
         |
         |
         v
+--------+--------+    +-----------------+
|                 |    |                 |
|    Username     +--->+   Send Error    |
|    Exists?      | No |    Message      |
|                 |    |                 |
+--------+--------+    +-----------------+
         |
     Yes |
         v
+--------+--------+    +-----------------+
|                 |    |                 |
|  User Deleted?  +--->+   Send Error    |
|                 | Yes|    Message      |
+--------+--------+    |                 |
         |             +-----------------+
      No |
         v
+--------+--------+    +-----------------+
|                 |    |                 |
|  User Banned?   +--->+   Send Error    |
|                 | Yes|    Message      |
+--------+--------+    |                 |
         |             +-----------------+
      No |
         v
+--------+--------+    +-----------------+
|                 |    |                 |
|   Login Rate    +--->+   Send Error    |
| Limit Reached?  | Yes|    Message      |
|                 |    |                 |
+--------+--------+    +-----------------+
         |
      No |
         v
+--------+--------+    +-----------------+
|                 |    |                 |
|Password Matches?+--->+   Send Error    |
|                 | No |    Message      |
+--------+--------+    |                 |
         |             +-----------------+
     Yes |
         v
+--------+--------+
|                 |
|   Login User    |
|                 |
+--------+--------+
         |
         |
         v
+--------+--------+
|                 |
|  Send Success   |
|    Message      |
|                 |
+-----------------+

Something should immediately stand out here: there are no enclosed regions in this diagram! Furthermore, even our new diagram is much simpler to follow than the old one was.

Reality is rarely simple.

The above is a really forgiving example. It has no loops, and loops are going to create enclosed regions that can't be broken apart so easily; it has no conditional branches that are so tightly coupled with the main path of execution that they can't be broken up; and the scope of functionality and side effects are minimal. Sometimes you can't break those regions up. So what do we do when we inevitably encounter these cases?

High cyclomatic complexity in your program as a whole is inevitable for sufficiently large projects, especially in a production environment, and your efforts to reduce it can only go so far. In fact, I don't recommend trying to remove all or even most instances of cyclomatic complexity at all--instead, you should just be keeping the concept in mind to determine whether or not a function, method, class, module, or other component of your system is accumulating technical debt and therefore in need of refactoring.

At this point, astute readers might ask, "How does refactoring help if the cyclomatic complexity doesn't actually go away?", and this is a valid concern. The answer to that is simple, however: we're hiding complexity behind abstractions.

To test this, let's forget about cyclomatic complexity for a moment and instead focus on simplifying the refactored version of our toy example using abstraction:

<?php

function handleLoginAttempt($login_data) {
    if(!usernameExists($login_data['username'])) {
        sendErrorResponse(getBadUsernameError($login_data['username']));
        return;
    }

    $user = getUser($login_data['username']);
    if(isDeleted($user)) {
        sendErrorResponse(getUserDeletedError($user));
        return;
    }

    if(isBanned($user)) {
        sendErrorResponse(getUserBannedError($user));
        return;
    }

    if(loginRateLimitReached($user)) {
        logBadLoginAttempt($user);
        sendErrorResponse(getLoginRateLimitError($user));
        return;
    }

    if(!passwordMatches($user, $login_data['password'])) {
        logBadLoginAttempt($user);
        sendErrorResponse(getBadPasswordError());
        return;
    }

    loginUser($user);
    sendSuccessResponse();
}

$login_data = getLoginCredentialsFromInput();

handleLoginAttempt($login_data);

?>

The code above is functionally identical to our refactored example from earlier, but has an additional abstraction via a function. Now we can diagram this higher-level abstraction as follows:

+-----------------+
|                 |
|  Program Start  |
|                 |
+--------+--------+
         |
         |
         v
+--------+--------+
|                 |
|  Attempt Login  |
|                 |
+-----------------+

This is, of course, a pretty extreme example, but this is how we handle thinking about complex program logic. We abstract it down to the barest basics so that we can visualize, in its simplest form, what the program is supposed to do. We don't actually care about the implementation unless we're digging into that specific part of the system, because otherwise we would be so bogged down by the details that we wouldn't be able to reason about what our program is supposed to do.

Likewise, we can use these abstractions to hide away the cyclomatic complexity underlying different components of our software. This keeps everything clean and clutter-free in our head. And the more we do to keep our smaller components simple and easy to think about, the easier the larger components are to deal with, no matter how much cyclomatic complexity all of those components share as a collective.

Final Thoughts

Cyclomatic complexity isn't a bad thing to have in your code. The concept itself is only intended to be used as one of many tools to assess when your code is accumulating too much technical debt. It's a warning sign that you may need to change something, nothing more. But it's an incredibly useful tool to have available to you and you should get comfortable using it.

As a general rule of thumb, you can usually just take a glance at your code and assess whether or not there's too much cyclomatic complexity in a component by looking for either of the following:

• Too many loops and/or conditional statements nested within each other, i.e. you have a lot of indentation.
• Many loops in the same function/method.

It's not a perfect rule of thumb, but it's useful for at least 90% of your development needs, and there will inevitably be cases where you will prefer to accept some greater cyclomatic complexity because there is some benefit that makes it a better trade-off. Making that judgment is up to you as a developer.

As always, I'm more than willing to listen to feedback and answer any questions!

Preface

Software security is one of those subjects that often gets overlooked, both in academia and in professional projects, unless you're specifically working with some existing security-related element (e.g. you're taking a course on security basics, or updating your password hashing algorithm). As a result, we frequently see stories of rather catastrophic data leaks from otherwise reputable businesses, leaks which should have been entirely preventable with even the most basic of safeguards in place.

With that in mind, I thought I would switch things up and discuss something security-related this time.

Background

It's commonplace for complex software systems to avoid unnecessarily large expenses, especially in terms of technical debt and the capital involved in the initial development costs of building entire systems for e.g. geolocation or financial transactions. Instead of reinventing the wheel and effectively building a parallel business, we instead integrate with existing third-party systems, typically by using an API.

The problem, however, is that sometimes these third-party systems process requests over a long period of time, potentially on the order of minutes, hours, days, or even longer. If, for example, you have users who want to purchase something using your online platform, then it's not a particularly good idea to having potentially thousands of open connections to that third-party system all sitting there waiting multiple business days for funds to clear. That would just be stupid. So, how do we handle this in a way that isn't incredibly stupid?

There are two commonly accepted methods to avoid having to wait around:

1. We can periodically contact the third-party system and ask for the current status of a request, or
2. We can give the third-party system a way to contact us and let us know when they're finished with a request.

Both of these methods work, but obviously there will be a potentially significant delay in #1 between when a request finishes and when we know that it has finished (with a maximum delay of the wait time between status updates), whereas in #2 that delay is practically non-existent. Using #1 is also incredibly inefficient due to the number of wasted status update requests, whereas #2 allows us to avoid that kind of waste. Clearly #2 seems like the ideal option.

Method #2 is what we call a webhook.

May I see your ID?

The problem with webhooks is that when you're implementing one, it's far too easy to forget that you need to restrict access to it. After all, that third-party system isn't a user, right? They're not a human. They can't just give us a username and password like we want them to. They don't understand the specific requirements for our individual, custom-designed system.

But what happens if some malicious actor figures out what the webhook endpoint is? Let's say that all we do is log webhook requests somewhere in a non-capped file or database table/collection. Barring all other possible attack vectors, we suddenly find ourselves susceptible to that malicious actor sending us thousands, possibly millions of fraudulent data payloads in a small amount of time thanks to a botnet, and now our server's I/O utilization is spiking and the entire system is grinding to a halt--we're experiencing a DDoS!

We don't want just anyone to be able to talk to our webhook. We want to make sure that anyone who does is verified and trusted. But since we can't require a username and password, since we can't guarantee that the third-party system will even know how to make use of them, what can we do?

The answer is to use some form of token-based authentication--we generate a unique token, kind of like an ID card, and we attach it to our webhook endpoint (e.g. https://example.com/my_webhook/{unique_token}). We can then check that token for validity every time someone touches our webhook, ensuring that only someone we trust can get in.

Class is in Session

Just as there are two commonly accepted models for how to handle receiving updates from third-party systems, there are also two common models for how to assign a webhook to those systems:

1. Hard-coding the webhook in your account settings, or
2. Passing a webhook as part of request payload.

Model #1 is, in my experience, the most common of the two. In this model, our authentication token is typically directly linked to some user or user-like object in our system. This token is intended to be persisted and reused indefinitely, only scrapped in the event of a breach or a termination of integration with the service that uses it. Unfortunately, if the token is present within the URL, it's possible for your token to be viewed in plaintext in your logs.

In model #2, it's perfectly feasible to mirror the behavior of model #1 by simply passing the same webhook endpoint with the same token in every new request; however, there is a far better solution. We can, instead, generate a brand new token for each new request to the third-party system, and each new token can be associated with the request itself on our own system. Rather than only validating the token itself, we then validate that the token and the request it's supposed to be associated with are both valid. This ensures that even in the event of a breach, a leaked authentication token's extent of damage is limited only to the domain of the request it's associated with! In addition, we can automatically expire these tokens after receiving a certain number of requests, ensuring that a DDoS using a single valid token and request payload isn't possible. As with model #1, however, we still run into problems of token exposure if the token is present in the URL.

Model #2 treats each individual authentication token not as a session for an entire third-party system, but as a session for a single request on that system. These per-request session tokens require greater effort to implement, but are inherently safer due to the increased granularity of our authentication and our flexibility in allowing ourselves to expire the tokens at will.

Final Thoughts

Security is hard. Even with per-request session tokens, webhooks still aren't as secure as we might like them to be. Some systems allow us to define tokens that will be inserted into the request payload, but more often than not you'll find that only a webhook URL is possible to specify. Ideally we would stuff those tokens right into the POST request payload for all of our third-party systems so they would never be so easily exposed in plaintext in log files, but legacy systems tend to be slow to catch up and newer systems often don't have developers with the security background to consider it.

Still, as far as securing webhooks goes, having some sort of cryptographically secure authentication token is far better than leaving the door wide open for any script kiddie having a bad day to waltz right in and set the whole place on fire. If you're integrating with any third-party system, your job isn't to make it impossible for them to get their hands on a key, but to make it really difficult and to make sure you don't leave any gasoline lying around in case they do.

Preface

Different projects have different use cases that can ultimately result in common solutions not suiting your particular needs. Today I'm going to diverging a bit from my more abstract, generalized topics on code quality and instead focus on a specific project structure example that I encountered.

Background

For a while now, I've found myself being continually frustrated with the state of my project configuration management. I had a single configuration file that would contain all of the configuration options for the various tools I've been using--database, API credentials, etc.--and I kept running into the problem of wanting to test these tools locally while not inadvertently committing and pushing sensitive credentials upstream. For me, part of my security process is ensuring that sensitive access credentials never make it into the repository and to limit access to these credentials to only people who need to be able to access them.

Monolithic Files Cause Monolithic Pain

The first thing I realized was that having a single monolithic configuration file was just terrible practice. There are going to be common configuration options that I want to have in there with default values, such as local database configuration pointing to a database instance running on the same VM as the application. These should always be in the repo, otherwise any dev who spins up an instance of the VM will need to manually tread documentation and copy-paste the missing options into the configuration. This would be incredibly time-consuming, inefficient, and stupid.

I also use different tools which have different configuration options associated with them. Having to dig through a single file containing configuration options for all of these tools to find the ones I need to modify is cumbersome at best. On top of that, having those common configuration options living in the same place that sensitive access credentials do is just asking for a rogue git commit -A to violate the aforementioned security protocol.

Same Problem, Different Structure

My first approach to resolving this problem was breaking the configuration out into separate files, one for each distinct tool. In each file, a "skeleton" config was generated, i.e. each option was given a default empty value. The main config would then only contain config options that are common and shared across the application. To avoid having the sensitive credentials leaked, I then created rules in the .gitignore to exclude these files.

This is where I ran into problem #2. I learned that this just doesn't work. You can either have a file in your repo and have all changes to that file tracked, have the file in your repo and make a local-only change to prevent changes from being tracked, or leave the file out of the repo completely. In my use case, I wanted to be able to leave the file in the repo, treat it as ignored by everyone, and only commit changes to that file when there was a new configuration option I wanted added to it. Git doesn't support this use case whatsoever.

This problem turned out to be really common, but the solution suggested is to have two separate versions of your configuration--one for dev, and one for production--and to have a flag to switch between the two. Given the breaking up of my configuration, I would then need twice as many files to do this, and given my security practices, this would violate the no-upstream rule for sensitive credentials. Worse still, if I had several different kinds of environments with different configuration--local dev, staging, beta, production--then for m such environments and n configuration files, I would need to maintain n*m separate files for configuration alone. Finally, I would need to remember to include a prefix or postfix to each file name any time I needed to retrieve values from a new config file, which is itself an error-prone requirement. Overall, there would be a substantial increase in technical debt. In other words, this approach would not only not help, it would make matters worse!

Borrowing From Linux

After a lot of thought, an idea occurred to me: within Linux systems, there's an /etc/skel/ directory that contains common files that are copied into a new user's home directory when that user is created, e.g. .bashrc and .profile. You can make changes to these files and have them propagate to new users, or you can modify your own personal copy and leave all other new users unaffected. This sounds exactly like the kind of behavior I want to emulate!

Following their example, I took my $APPHOME/config/ directory and placed a skel/ subdirectory inside, which then contained all of the config files with the empty default values within. My .gitignore then looked something like this:

$APPHOME/config/*
!$APPHOME/config/main.php
!$APPHOME/config/skel/
!$APPHOME/config/skel/*
# This last one might not be necessary, but I don't care enough to test it without.

Finally, on deploying my local environment, I simply include a snippet in my script that enters the new skel/ directory and copies any files inside into config/, as long as it doesn't already exist:

cd $APPHOME/config/skel/
for filename in *; do
    if [ ! -f "$APPHOME/config/$filename" ]; then
        cp "$filename" "$APPHOME/config/$filename"
    fi
done

(Note: production environments have a slightly different deployment procedure, as local copies of these config files are saved within a shared directory for all releases to point to via symlink.)

All of these changes ensure that only config/main.php and the files contained within config/skel/ are whitelisted, while all others are ignored, i.e. our local copies that get stored within config/ won't be inadvertently committed and pushed upstream!

Final Thoughts

Common solutions to problems are typically common for a good reason. They're tested, proven, and predictable. But sometimes you find yourself running into cases where the common, well-accepted solution to the problem doesn't work for you. Standards exist to solve a certain class of problems, and sometimes your problem is just different enough for it to matter and for those standards to not apply. Standards are created to address most cases, but edge cases will always exist. In other words, standards are guidelines, not concrete rules.

Sometimes you need to stop thinking about the problem in terms of the standard approach to solving it, and instead break it down into its most abstract, basic form and look for parallels in other solved problems for inspiration. Odds are the problem you're trying to solve isn't as novel as you think it is, and that someone has probably already solved a similar problem before. Parallels, in my experience, are usually a pretty good indicator that you're on the right track.

More importantly, there's a delicate line to tread between needing to use a different approach to solving an edge case problem you have, and needing to restructure your project to eliminate the edge case and allow the standard solution to work. Being able to decide which is more appropriate can have long-lasting repercussions on your ability to manage technical debt.

Preface

Occasionally I feel the need to touch on the subject of code quality, particularly because of the importance of its impact on technical debt, especially as I continue to encounter the effects of technical debt in my own work and do my best to manage it. It's a subject that is unfortunately not emphasized nearly enough in academia.

Background

As a refresher, technical debt is the long-term cost of the design decisions in your code. These costs can manifest in different ways, such as greater difficulty in understanding what your code is doing or making non-breaking changes to it. More generally, these costs manifest as additional time and resources being spent to make some kind of change.

Sometimes these costs aren't things you think to consider. One such consideration is how difficult it might be to upgrade a specific technology in your stack. For example, what if you've built a back-end system that integrates with AWS and you suddenly need to upgrade your SDK? In a small project this might be easy, but what if you've built a system that you've been maintaining for years and it relies heavily on AWS integrations? If the method names, namespaces, argument orders, or anything else has changed between versions, then suddenly you'll need to update every single reference to an AWS-related tool in your code to reflect those changes. In larger software projects, this could be a daunting and incredibly expensive task, spanning potentially weeks or even months of work and testing.

That is, unless you keep those references to a minimum.

A Toy Example

This is where "wrapping" your external libraries comes into play. The concept of "wrapping" basically means to create some other function or object that takes care of operating the functions or object methods that you really want to target. One example might look like this:

<?php

class ImportedClass {
    public function methodThatMightBecomeModified($arg1, $arg2) {
        // Do something.
    }
}

class ImportedClassWrapper {
    private $class_instance = null;

    private function getInstance() {
        if(is_null($this->class_instance)) {
            $this->class_instance = new ImportedClass();
        }

        return $this->class_instance;
    }

    public function wrappedMethod($arg1, $arg2) {
        return $this->getInstance()->methodThatMightBecomeModified($arg1, $arg2);
    }
}

?>

Updating Tools Doesn't Have to Suck

Imagine that our ImportedClass has some important new features that we need to make use of that are only available in the most recent version, and we're several versions behind. The problem, of course, is that there were a lot of changes that ended up being made between our current version and the new version. For example, ImportedClass is now called NewImportedClass. On top of that, methodThatMightBecomeModified is now called methodThatWasModified, and the argument order ended up getting switched around!

Now imagine that we were directly calling new ImportedClass() in many different places in our code, as well as directly invoking methodThatMightBecomeModified:

<?php

$imported_class_instance = new ImportedClass();
$imported_class_instance->methodThatMightBeModified($val1, $val2);

?>

For every single instance in our code, we need to perform a replacement. There is a linear or--in terms of Big-O notation--a complexity of O(n) to make these replacements. If we assume that we only ever used this one method, and we used it 100 times, then there are 100 instances of new ImportClass() to update and another 100 instances of the method invocation, equaling 200 lines of code to change. Furthermore, we need to remember each of the replacements that need to be made and carefully avoid making any errors in the process. This is clearly non-ideal.

Now imagine that we chose instead to use the wrapper object:

<?php

$imported_class_wrapper = new ImportedClassWrapper();
$imported_class_wrapper->wrappedMethod($val1, $val2);

?>

Our updates are now limited only to the wrapper class:

<?php

class ImportedClassWrapper {
    private $class_instance = null;

    private function getInstance() {
        if(is_null($this->class_instance)) {
            $this->class_instance = new NewImportedClass();
        }

        return $this->class_instance;
    }

    public function wrappedMethod($arg1, $arg2) {
        return $this->getInstance()->methodThatWasModified($arg2, $arg1);
    }
}

?>

Rather than making changes to 200 lines of code, we've now made changes to only 2. What was once an O(n) complexity change has now turned into an O(1) complexity change to make this upgrade. Not bad for a few extra lines of code!

A Practical Example

Toy problems are all well and good, but how does this translate to reality?

Well, I ran into such a problem myself once. Running MongoDB with PHP requires the use of an external driver, and this driver provides an object representing a MongoDB ObjectId. I needed to perform a migration from one hosting provider over to a new cloud hosting provider, with the application and database services, which were originally hosted on the same physical machine, hosted on separate servers. For security reasons, this required an upgrade to a newer version of MongoDB, which in turn required an upgrade to a newer version of the driver.

This upgrade resulted in many of the calls to new MongoId() failing, because the old version of the driver would accept empty strings and other invalid ID strings and default to generating a new ObjectId, whereas the new version of the driver treated invalid ID strings as failing errors. And there were many, many cases where invalid strings were being passed into the constructor.

Even after spending hours replacing the (literally) several dozen instances of the constructor calls, there were still some places in the code where invalid strings managed to get passed in. This made for a very costly upgrade.

The bugs were easy to fix after the initial replacements, though. After wrapping new MongoId() inside of a wrapper function, a few additional conditional statements inside of the new function resolved the bugs without having to dig around the rest of the code base.

Final Thoughts

This is one of those lessons that you don't fully appreciate until you've experienced the technical debt of an unwrapped external library first-hand. Code quality is an active effort, but a worthwhile one. It requires you to be willing to throw away potentially hours or even days of work when you realize that something needs to change, because you're thinking about how to keep yourself from banging your head against a wall later down the line instead of thinking only about how to finish up your current task.

"Work smarter, not harder" means putting in some hard work upfront to keep your technical debt under control.

That's all for now, and remember: don't be fools, wrap your external tools.

Preface

In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer productivity, and the need to pay down on that debt. Today I would like to touch on a practical example that I'd encountered in a production environment.

Background

Before we can discuss the refactor itself, it's important to be on the same page regarding the technologies being used. In my case, I work with PHP utilizing a proprietary back-end framework and MongoDB as our database.

PHP is a server-side scripting language. Like many scripting languages, it's loosely typed. This has some benefits and drawbacks.

MongoDB is a document-oriented database. By default it's schema-less, allowing you to make any changes at will without an update to schema. This can blend pretty well with the loose typing of PHP. Each document is represented using a JSON-like structure and is stored in something called a "collection". For those of you accustomed to using relational database, a "collection" is analogous to a table, each document is a row, and each field in the document is a column. A typical query in the MongoDB shell would look something like this:

db.users.findOne({
    username: "Emerald_Knight"
});

The framework itself has some framework-specific objects that are held in global references. This makes them easily accessible, but naturally littering your code with a bunch of globals is both error-prone and an eyesore.

Unexpected Spaghetti

In my code base are a number of different objects that are designed to handle basic CRUD-like operations on their associated database entries. Some of these objects hold references to other objects, so naturally there is some data validation that occurs to ensure that the references are both valid and authorized. Pretty typical stuff.

What I noticed, however, is that the collection names for these database entries were littered throughout my code. This isn't necessarily a bad thing, except there were some use cases that came to mind: what if it turned out that my naming for one or more of these collections wasn't ideal? What if I wanted to change a collection name for the sake of easier management on the database end? What if I have a tendency to forget the name of a database collection and constantly have to look it up? What if I make a typo of all things? On top of that, the framework's database object was stored in a global variable.

These seemingly minor sources of technical debt end up adding up over time and could cause some serious problems in the worst case. I've had breaking bugs make their way passed QA in the past, after all.

Exchanging Spaghetti for Some Light Lasagna

The problem could be characterized simply: there were scoping problems and too many references to what were essentially magic strings. The solution, then, was to move the database object reference from global to local scope within the application code and to eliminate the problem of magic strings. Additionally, it's a good idea to avoid polluting the namespace with an over-reliance on constants, and using those constants for database calls can also become unsightly and difficult to follow as those constants could end up being generally disconnected from the objects they're associated with.

There turned out to be a nice, object-oriented, very PHP-like solution to this problem: a so-called "magic method" named "__call". This method is invoked whenever an "inaccessible" method is called on the object. Using this method, a database command executed on a non-database object could pass the command to the database object itself. If this logic were placed within an abstract class, the collection could then be specified simply as a configuration option in the inheriting class.

This is what such a solution could look like:

<?php

abstract class MyBaseObject {

    protected $db = null;
    protected $collection_name = null;

    public function __construct() {
        global $db;
        
        $this->db = $db;
    }

    public function __call($method_name, $args) {
        if(method_exists($this->db, $method_name)) {
            return $this->executeDatabaseCommand($method_name, $args);
        }

        throw new Exception(__CLASS__ . ': Method "' . $method_name . '" does not exist.');
    }

    public function executeDatabaseCommand($command, $args) {
        $collection = $this->collection_name;
        $db_collection = $this->db->$collection;

        return call_user_func_array(array($db_collection, $command), $args);
    }
}

class UserManager extends MyBaseObject {
    protected $collection_name = 'users';

    public function __construct() {
        parent::__construct();
    }
}

$user_manager = new UserManager();
$my_user = $user_manager->findOne(array('username'=>'Emerald_Knight'));

?>

This solution utilizes a single parent object which transforms a global database object reference into a local one, eliminating the scope issue. The collection name is specified as a class property of the inheriting object and only used in a single place in the parent object, eliminating the magic string and namespace polluting issues. Any time you perform queries on users, you do so by using the UserManager class, which guarantees that you will always know that your queries are being performed on the objects that you intend. And finally, if the collection name for an object class ever needs to be updated, it's a simple matter of modifying the single instance of the class property $collection_name, rather than tracking down some disconnected constant.

Limitations

This, of course, doesn't solve all of the existing problems. After all, executing the database queries for one object directly from another is still pretty bad practice, violating the principle of separation of concerns. Instead, those queries should generally be encapsulated within object methods and the objects themselves given primary responsibility in handling associated data. It's also incredibly easy to inadvertently override a database method, e.g. defining a findOne() method on UserManager, so there's still some mindfulness required on the part of the programmer.

Still, given the previous alternative, this is a pretty major improvement, especially for an initial refactor.

Final Thoughts

As always, technical debt is both necessary and inevitable. After all, in exchange for not taking the excess time and considering structuring my code this way in the beginning, I had greater initial velocity to get the project off of the ground. What's important is continually reviewing your code as you're building on top of it so that you can identify bottlenecks as they begin to strain your efficiency, and getting those bottlenecks out of the way.

In other words, even though technical debt is often necessary and is certainly inevitable, it's important to pay down on some of that debt once it starts getting expensive!

Preface

One of the most common questions I see from prospective programmers and computer scientists is "where should I start?". My answer to that is a pretty consistent one: learn how to solve problems effectively. But that's vague and not really all that helpful, so I figured that I should actually tackle this in a little more depth by touching on something more specific.

Specifically, I want to touch on the subject of how to think about complex problems.

The Rationale Behind Learning

Before we can better understand how to effectively solve problems, it's important to consider how it is that we learn. With any subject, the standard approach is to begin with the bare basics. For programming, that's writing a Hello, World! program in the new language you're working with. For foreign languages, you learn basic common words and sentence structure. For math, you learn your basic arithmetic operations like addition and multiplication.

From there, we add on more additional complexity and string together everything we've learned. For a foreign language, this looks like learning about new words, stringing them together in your own sentences, then learning about verb tenses and throwing them into the mix as well. With math, you take your normal number crunching and suddenly throw the concept of order of operations into the mix, then variables and how to solve for them.

As a general rule, we first get comfortable with solving a simple problem and gradually build up toward solving increasingly more difficult ones.

The Missing Piece

Odds are that we've all sat in a math class at one point, and when the teacher asked a student how to solve a problem, they received an immediate "I don't know". You may or may not have been that kid yourself. I have no intention of shaming the kids who struggled (or those who still struggle) with math. Rather, I want to point to what I believe is the fundamental cause of that mental barrier that has frustrated students for generations.

Learning is not simply a matter of adding more complexity to problems. A key part of learning, and one that I don't recall ever having emphasized during my grade school studies, is your ability to break problems down into the steps that you know how to complete and combine the different, simpler skills you've already learned to arrive at a solution. Instead, you were expected to solve many of those complex problems and learn through practice, or through pure rote memorization.

What determined whether or not you could solve those problems was then a question of whether or not you could intuit or memorize how to solve those specific problems, and brand new problems that still made use of the same skill sets but had completely different forms would throw a wrench in that. Those who could solve any of those problems--those who, I would argue, were often mistakenly referred to as "geniuses" or "talented"--were really just those who knew how to break a problem down into simpler pieces.

This isn't a failing on the students, but on the way they've been taught to think about problems.

Reducing Problems

What does it mean to "break down" a problem, though? The few times I recall a teacher ever touching on the subject, "break down the problem" and "use the skills you've already learned" were the kinds of pieces of advice passed around, completely vague and devoid of meaning for anyone who didn't already understand. How can we better grasp this important step?

There's a term in complexity theory known as "reduction". The general idea is that if you have problems A and B, where you already know how to solve B, then if you can transform problem A so that it looks like problem B, then you can use your solution for B to solve at least part of A.

In other words, finding the solution to a more complex problem is just a matter of finding a way to make it look like a problem you already know how to solve.

The advice to "break down" a problem really means to perform this process of "reduction", of transforming your more complicated problem A into your simpler, known problem B.

In Practice

We're still discussing a vague concept, but now that we have more specific language to work with, we can more easily see how it works in practice (a reduction of its own!).

Let's consider a conceptually simple problem: grabbing the k^th largest (or smallest) item from a list. How do we solve this problem? Probably the most obvious and straightforward answer is to sort the list then grab the k^th item, right?

Notice that we gave two high-level descriptions of the steps we need to solve this problem: sorting, then grabbing the appropriate item. We can therefore then state that the problem of "grab the k^th largest/smallest item from a list" can be reduced to the two problems "sort a list" and "grab the k^th item from a list".

Now, let's say we're given the problem "take this list of competitor times from the race and tell me what the top 10 race times were". What do we know about this problem? We know that we're being given a list, and we know that we need the 10 smallest items from that list. We also know that "10 smallest items" is just shorthand for "the 1st smallest item, the 2nd smallest item, ..., and the 10th smallest item". We can therefore reduce this problem to the previous one we solved by transforming it into "grab the k^th smallest item from a list" and "repeat for values 1-10 for k".

Practical Advice

In the end, my explanation may not have helped much at all in actually grasping the concept of reduction. My intent isn't necessarily to help you understand it immediately, but to provide you a framework for a way of thinking. Even if you do grasp the general concept, you may even wonder how you're supposed to recognize these kinds of reductions out in the wild in non-academic environments. The answer, perhaps annoying, is practice. Much like an appraiser can only become good at discerning details through experience, a programmer or computer scientist can only recognize these patterns through repeated exposure.

In general, if I had to narrow it down to a small list of tips for improving your problem solving skills, this would be it:

• Work on grasping the concept of reduction itself.
• Expose yourself to lots of new problems.
• Don't shy away from difficult problems. Reduce them as much as you can and solve the pieces you're able to. Try to research the pieces you're struggling with. Return to the problem later when you have more experience if you have to, but take a crack at it first.
• Don't accept "I don't know" as an answer in itself. Ask yourself why you don't how to solve a problem. Narrow down which pieces you're able to solve and which pieces you're not.
• Just solve problems. Any problems. Easy ones, hard ones, and anything in between. Solving problems is a skill, and practicing it will make you better at solving problems in general, and better at recognizing the simpler problems inside of more complicated ones.
• Don't just come up with a solution to a problem. Ensure that you understand how each piece of it works and why it works. Copy-pasting from StackOverflow can be a valid tool at your disposal, but doing so mindlessly isn't nearly as valuable as reviewing the solution, being able to determine whether or not it works before ever executing the code, and being able to discard anything unnecessary from it.

Final Thoughts

I'm not an authoritative voice on this subject. I'm not an educator. More than anything, I'm a life-long student and an enthusiast. There's seldom a day when I don't have to research something new in order to solve a problem I'm not familiar with, or remind myself the syntax for a function I've used several times in the past. I don't know anything about teaching others, but I do know plenty about learning, and if there's anything that has stood out to me over the years, it's the fact that I find it easier to learn about something or to solve a problem if I can transform the concept into something that's easier for me to grasp.

Moreover, I'm human and thus prone to mistakes. Call me out on them if you notice them. I'll take any of my mistakes as learning opportunities :)

Preface

Over the last couple of years, I've had the opportunity to learn from the mistakes of my predecessors and put those lessons into practice. Among those lessons, three have stood out to me in particular:

1. Consistency is king.
2. Try not to be too clever for your own good.
3. Good code takes time.

I know that there are a lot of new and aspiring programmers here (and I'm admittedly far from being a guru myself), so I thought it would be good to touch on these three lessons, what they mean, and why they're so important.

Consistency is King

This is something that I had drilled into my head over nearly two years working on the code base at my previous job. Not by my fellow programmers (who did not exist), nor by my boss, but by the code itself.

Consistency can mean a number of things, but there are two primary points that matter:

1. Syntactic consistency.
2. Architectural consistency.

Syntactic consistency concerns standards in what your code looks like. For example, the choice between snake_case or camelCase or PascalCase for naming; function parameter order; or even something as benign as what kind of indentation and how much of it you use.

Architectural consistency concerns standards in how you structure your code. Making sure that you either use public class properties or getter and setter methods; using multiple booleans or using bitmasks; using or not using objects for encapsulating data to be passed around; validating data within the primary object or relegating that responsibility to a validator class; and other seemingly minor decisions about how you handle certain behavior make a big difference.

The code base I maintained had no such consistency. You could never remember whether the method you needed to call was named using snake_case or camelCase and had to perform several searches just to find it. Worse still, some methods defined to handle Ajax calls were prefixed with ajax while many weren't. Argument ordering seemed to be determined by a coin flip, and indentation seemed to vary between 2-space, 3-space, 4-space, and even 5-space indentation depending on what mood my predecessor was in at the time. You often could not tell where a function's body began and where it ended. Writing code was an exercise both in problem solving and in deciphering ancient religious texts.

Architecturally it was no better. There was no standardization in how data was validated or sanitized, how class members were accessed or modified, how functionality was inherited, whether the functionality was encapsulated in an object method or in a function, or which objects were responsible for which behavior.

That lack of consistency makes introducing or modifying a small feature, a task which should ordinarily be a breeze, an engineering feat of its own. Often you end up implementing that feature, after dancing around the tangled mess of spaghetti, only to find that the functionality that you implemented already existed somewhere else in the code base but was hiding out in a deep, dark corner that you never even knew was there until you had to fix some other broken feature months later and happened to stumble across it.

Consistency means predictability, and predictability means discoverability and, more importantly, easier changes and higher confidence in those changes.

Cleverness is a Fallacy

In any given project, it can be tempting to do something that saves you extra lines of code, or saves on CPU cycles, or just looks awesome and does something nobody would have thought of before. As human beings and especially as craftsmen, we like to leave our mark and take pride in breaking the status quo by taking a novel and interesting approach to a problem. It can make us feel fulfilled in our work, that we've done something unique, a trademark of sorts.

The problem with that is that it directly conflicts with the aforementioned consistency and predictability. What ends up being an engineering wonder to you ends up being an engineering nightmare to someone else. While you're enjoying the houses you build with wall studs arranged in the shape of a spider's web, the home remodelers who come along later aren't even sure if they can change part of the structure without causing the entire wall to collapse, and they're not even sure which walls are load-bearing and which aren't, so they're basically playing Jenga while blindfolded.

The code base I maintained had a few such gems, with what looked like load-bearing walls but were actually made of papier-mâché and were only decorative in nature, and the occasional spider's web wall studs. One spider's web comes to mind in particular. It's been a while since I've worked on that piece of code, so I can't recall what exactly it did, but two query-constructing pieces of logic had overlapping query structure with the difference being the operators and data. Rather than being smart and allowing those two constructs to be different, however, my predecessor decided to be clever and the query construction was abstracted into a separate method so that the same general query structure could be used in other places (note: it never was, and was only ever used in those two instances). It was abstracted so that all original context was lost and no comments existed to explain any of it. On top of that, the method was being called from the most critical piece of the system which, unfortunately, was already a convoluted mess and desperately required a rewrite and thus required me to understand what the hell that method was even doing (incidentally, I fell in love with whiteboards as a result).

When you feel like you're being clever, you should always stop what you're doing and make sure that what you're doing isn't actually a really terrible idea. Cleverness doesn't exist. Knowledge and intelligence do. Write intelligent code, not clever code.

Good Code Takes Time

Bad code more often than not is the result of impatience. We don't like to plan out the solution before we get to writing code. We like to use variables like x and temp in order to quickly achieve functional correctness of our code because stopping to think about how to name them is just additional overhead getting in the way. We don't like to scrap our bad work if we can salvage it in some way instead, because then we have to start from scratch and that's daunting. We continually work against ourselves and gradually increase our mental overhead because we try to decrease our mental overhead. As a result we find ourselves too exhausted by the end of our initial implementations to concern ourselves with fixing obvious problems. Obviously bad but functional code is preferable because we just want the task to be done and over with.

The more you get exposed to bad code and the more you try to avoid pushing that hell onto yourself and your successors, the more you realize that you need to spend less time coding and more time researching and planning. Whereas you may have been spending upwards of 50% of your time coding previously, suddenly you find yourself spending as little as 10% of your time writing any code at all.

Professionals from just about any field can tell you that you can either do something right or you can do it twice. You might recognize this most easily in the age-old piece of woodworking wisdom, "measure twice, cut once". The same is true of code, and doing something right means planning how to do it right in the first place before you've even started on the job.

Putting into Practice

I've been fortunate over the last couple of months to be able to start on a brand new project and architect it in a way that I see fit. Changes which would ordinarily take days or weeks in the old code base now take me half a day at most, and a matter of minutes at best. I remember where to find a piece of code that I need because I'm consistent and predictable about where I place things; I don't struggle to tell where something begins and where it ends because I'm consistent about structure; I don't continually hate myself when I need to make changes to my code because I don't do anything wildly out of the ordinary; and most importantly, I take my time to figure out what it is that I need to do and how I want to do it before I've written a single line of code.

When I needed to add a web portal interface for uploading a media asset to associate with a database object, the initial implementation took me a week, due to the need for planning, adding the interface, and supporting and debugging the asset management. When I needed to extended that interface to allow for uploading the same kinds of assets for a completely different object type, it took me only half an hour, with most of that time being dedicated toward updating a Vue.js component to accept configuration via props rather than working for only the single hard-coded object type. If I need to add a case for any additional object type, it will take me only five minutes.

That initial week of work for the web interface provided me with cost savings that would not have been feasible otherwise, and that initial week of work would have taken as many as three weeks had I not structured the API to be as consistent as it is now. Every initial lag in implementation is offset heavily by the long-term cost savings of writing good code.

Technical Debt

Technical debt is the cost of your code over time. The messier and worse your code gets, the more it costs you to try to change, and those costs only build up. Even good code can accumulate technical debt if the needs for your software have changed and its current architecture isn't compatible with those changes.

No project is without technical debt. Even my own code, that I've been painstakingly working on for the last couple of months, has technical debt. Odds are a programmer far more experienced than I am will come along and want to scrap everything I've done, and will do a far better job rewriting it.

That's okay, though. In fact, a certain amount of technical debt is good. If we try to never write any bad code whatsoever, then we could never possibly get to writing any code at all, because there are far too many unknowns for a new project.

What's important is knowing when to pay down on that technical debt, which could mean anything from paying it up front (i.e. through planning ahead of time) to paying it down when it starts to get too expensive (e.g. refactoring a complicated section of code when changes become sufficiently difficult). That's not something you can learn through a StackOverflow post or a college lecture, and certainly not from some unknown stranger on some relatively unknown website in a long, informal blog-like post.

Final Thoughts

I'm far from being a great programmer. There's a lot that I don't know and I still have quite a bit to learn. I love programming, though, and more than that I enjoy sharing the lessons I've learned with others. Especially the ones that I wish I'd learned back in college.

Please feel free to share your own experiences, learned lessons, and (if you have it) feedback here. I'd love to read up on some other thoughts on this subject!