• Activity
  • Votes
  • Comments
  • New
  • All activity
    1. Light Analysis of a Recent Code Refactor

      Preface In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer...

      Preface

      In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer productivity, and the need to pay down on that debt. Today I would like to touch on a practical example that I'd encountered in a production environment.


      Background

      Before we can discuss the refactor itself, it's important to be on the same page regarding the technologies being used. In my case, I work with PHP utilizing a proprietary back-end framework and MongoDB as our database.

      PHP is a server-side scripting language. Like many scripting languages, it's loosely typed. This has some benefits and drawbacks.

      MongoDB is a document-oriented database. By default it's schema-less, allowing you to make any changes at will without an update to schema. This can blend pretty well with the loose typing of PHP. Each document is represented using a JSON-like structure and is stored in something called a "collection". For those of you accustomed to using relational database, a "collection" is analogous to a table, each document is a row, and each field in the document is a column. A typical query in the MongoDB shell would look something like this:

      db.users.findOne({
          username: "Emerald_Knight"
      });
      

      The framework itself has some framework-specific objects that are held in global references. This makes them easily accessible, but naturally littering your code with a bunch of globals is both error-prone and an eyesore.


      Unexpected Spaghetti

      In my code base are a number of different objects that are designed to handle basic CRUD-like operations on their associated database entries. Some of these objects hold references to other objects, so naturally there is some data validation that occurs to ensure that the references are both valid and authorized. Pretty typical stuff.

      What I noticed, however, is that the collection names for these database entries were littered throughout my code. This isn't necessarily a bad thing, except there were some use cases that came to mind: what if it turned out that my naming for one or more of these collections wasn't ideal? What if I wanted to change a collection name for the sake of easier management on the database end? What if I have a tendency to forget the name of a database collection and constantly have to look it up? What if I make a typo of all things? On top of that, the framework's database object was stored in a global variable.

      These seemingly minor sources of technical debt end up adding up over time and could cause some serious problems in the worst case. I've had breaking bugs make their way passed QA in the past, after all.


      Exchanging Spaghetti for Some Light Lasagna

      The problem could be characterized simply: there were scoping problems and too many references to what were essentially magic strings. The solution, then, was to move the database object reference from global to local scope within the application code and to eliminate the problem of magic strings. Additionally, it's a good idea to avoid polluting the namespace with an over-reliance on constants, and using those constants for database calls can also become unsightly and difficult to follow as those constants could end up being generally disconnected from the objects they're associated with.

      There turned out to be a nice, object-oriented, very PHP-like solution to this problem: a so-called "magic method" named "__call". This method is invoked whenever an "inaccessible" method is called on the object. Using this method, a database command executed on a non-database object could pass the command to the database object itself. If this logic were placed within an abstract class, the collection could then be specified simply as a configuration option in the inheriting class.

      This is what such a solution could look like:

      <?php
      
      abstract class MyBaseObject {
      
          protected $db = null;
          protected $collection_name = null;
      
          public function __construct() {
              global $db;
              
              $this->db = $db;
          }
      
          public function __call($method_name, $args) {
              if(method_exists($this->db, $method_name)) {
                  return $this->executeDatabaseCommand($method_name, $args);
              }
      
              throw new Exception(__CLASS__ . ': Method "' . $method_name . '" does not exist.');
          }
      
          public function executeDatabaseCommand($command, $args) {
              $collection = $this->collection_name;
              $db_collection = $this->db->$collection;
      
              return call_user_func_array(array($db_collection, $command), $args);
          }
      }
      
      class UserManager extends MyBaseObject {
          protected $collection_name = 'users';
      
          public function __construct() {
              parent::__construct();
          }
      }
      
      $user_manager = new UserManager();
      $my_user = $user_manager->findOne(array('username'=>'Emerald_Knight'));
      
      ?>
      

      This solution utilizes a single parent object which transforms a global database object reference into a local one, eliminating the scope issue. The collection name is specified as a class property of the inheriting object and only used in a single place in the parent object, eliminating the magic string and namespace polluting issues. Any time you perform queries on users, you do so by using the UserManager class, which guarantees that you will always know that your queries are being performed on the objects that you intend. And finally, if the collection name for an object class ever needs to be updated, it's a simple matter of modifying the single instance of the class property $collection_name, rather than tracking down some disconnected constant.


      Limitations

      This, of course, doesn't solve all of the existing problems. After all, executing the database queries for one object directly from another is still pretty bad practice, violating the principle of separation of concerns. Instead, those queries should generally be encapsulated within object methods and the objects themselves given primary responsibility in handling associated data. It's also incredibly easy to inadvertently override a database method, e.g. defining a findOne() method on UserManager, so there's still some mindfulness required on the part of the programmer.

      Still, given the previous alternative, this is a pretty major improvement, especially for an initial refactor.


      Final Thoughts

      As always, technical debt is both necessary and inevitable. After all, in exchange for not taking the excess time and considering structuring my code this way in the beginning, I had greater initial velocity to get the project off of the ground. What's important is continually reviewing your code as you're building on top of it so that you can identify bottlenecks as they begin to strain your efficiency, and getting those bottlenecks out of the way.

      In other words, even though technical debt is often necessary and is certainly inevitable, it's important to pay down on some of that debt once it starts getting expensive!

      7 votes
    2. Let's talk best-practice Jenkins on AWS ECS

      [seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly] I've had some...

      [seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly]

      I've had some experience in this realm for a while now, but I'm having a little trouble with one issue in particular. Before I divulge, I'll present my thoughts on best practice and and what I've been able to implement:

      • Terraform everything (in accordance to terragrunt's "style guide" i.e. organization)
        THIS IS A BIG ONE: for the jenkins master task, make sure to use the following args to make sure jenkins jobs aren't super slow as hell to start:
      -Djava.awt.headless=true -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
      

      THIS IS A GAME CHANGER (more-so on k8s clusters when the ecs plugin isn't used... hint, it's shit).

      • Create an EFS (in a separate terraform module) and mount it to the jenkins ECS cluster at /var/jenkins_home. Makes jenkins much more reliable through outages and easier to upgrade.
      • Run a logging agent (via docker container) like logspout or newrelic or whatever IN USER_DATA and not as a task - that way you get logs if there are issues during user_data/cloud_init... this I'm actually not sure about. Running a container outside the context of an ECS task means the ECS agent can't really track it and allocate mem/cpu properly... but it does help with user_data triage.
      • Use pipelines and git plugins to drive jobs. All jenkins jobs should be in source control!
      • Make sure you setup docker cleanup jobs on DAY 1! If you hace limited access to your cluster and you run out of disk due to docker cache, networks, volumes, etc... you're screwed till the admin ssh's in and runs a prune. Get a docker system prune going or the equivalent for each docker resource with appropriate filters... i.e. filter for anything older than a few days and is dangling.
      • Use Jenkins Global Libraries to make Jenkinsfiles cleaner (I always just use vars instead of groovy/java style packages because it's easier and less ugly)
        Jenkinsfiles should mostly call other bash files, make files, python scripts to generate and load prop files, etc. The less logic you put in a Jenkinsfile (which is just modified groovy) the better. String interpolation, among other things, is a fuckery that we don't have time to triage.
      • (out-of-scope) Move to using k8s/EKS instead of ECS asap because the ECS plugin for jenkins is absolute shit and it doesn't use priority correctly (sorry whoever developed it and... oh wait abandoned it and hasn't merged anything for years... for for real it's cool, just give admin to someone else).
      • (cultural) Stop calling them slaves. "Hey @eng, we're rotating slaves due to some cache issues. If you have been affected by race conditions in that past, our new update and slave rotation should fix that. Our update may have killed your job that was running on an old slave, just wait a few and the new slaves will be ready" <--This just doesn't look good.
        Hope that was some good stuff for you guys. Maybe I'm preaching to the choir, but I've seen some pretty shit jenkins setups.

      NOW FOR MY QUESTION!

      Has ANYONE actually been able to setup a proper jenkins user on ECS that actually works for both a master and ephemeral jenkins-agents so that they can mount and use the docker.sock for builds without hitting permission issues? I'm talking using the ecs plugin and mounting docker.sock via that.

      I have always resorted to running jenkins master and agents as root, which means you have to chmod files (super expensive time and cpu for services with tons of files). Running microservices as root is obviously bad practice, and chmod-ing a zilliion files is shit for docker cache and time... so I want to get jenkins users able to utilize the docker.sock. THIS IS SPECIFICALLY FOR THE AWS ECS AMI! I don't care about debian or old versions of docker where you could use DOCKER_OPTS. That doesn't work on the AWS Linux image.

      Thanks! And happy Friday!

      5 votes
    3. Getting Started as a Developer from Scratch

      I have been interested in making the gradual career change to software development from my current humanities field. This stems from a handful of different places. Of course the pay and...

      I have been interested in making the gradual career change to software development from my current humanities field. This stems from a handful of different places. Of course the pay and flexibility are strong drivers but I like the idea of a field that is somewhat of a creative expression; one where you can manifest your knowledge and experience into something tangible.

      I have no experience with programming other than SQL use in ArcGIS and am hoping to gain some knowledge about the field; so anything would be helpful. Whether what to expect from this line of work, where someone with no experience should look to get started and what to expect, personal journeys, etc.

      Cheers!

      14 votes
    4. Breaking all the rules

      In most of my programming, I try and remain professional, and do things in a readable, maintainable way, that doesn't involve pushing the language to breaking point. But, occasionally, I give...

      In most of my programming, I try and remain professional, and do things in a readable, maintainable way, that doesn't involve pushing the language to breaking point.

      But, occasionally, I give myself free reign. What if you didn't care about the programmer who came after you? What if you didn't care about what a programmer should do in a certain circumstance?

      For myself, over the years I've written and rewritten a C library, I like to call CNoEvil. Here's a little taste of what you could do:

      #define EVIL_IO
      #define EVIL_COROUTINE
      #include "evil.h"
      
      proc(example, int)
        static int i = 0;
        coroutine();
        While 1 then
          co_return(++i);
        end
        co_end();
        return i;
      end
      
      Main then
        displayln(example());
        displayln(example());
      end
      

      (And yes, that compiles. Without warnings. Even with -Wpedantic.)

      So... Here's the challenge:

      Ignoring the rules, and best practices... How can you take your favourite programming language... And make it completely unrecogniseable?

      (Might be best to choose a language with macros, like Nim, Rust, any of the Lisps. Though, you can still do some impressively awful things in Java or Python, thanks to overloading inbuilt classes.)

      Challenge Ideas:

      • Make Python look like C
      • Make Java look like Python
      • Make anything look like BrainFuck

      I don’t know how to really explain my fascination with programming, but I’ll try. To somebody who does it, it’s the most interesting thing in the world. It’s a game much more involved than chess, a game where you can make up your own rules and where the end result is whatever you can make of it. - Linus Torvalds

      21 votes
    5. Any resources exploring the gap between beginner and top 5% expert in various tech fields?

      I am looking for any resource that could tell me how much knowledge and training is needed to go from a beginner to expert, in let's say application software development for example. Or in...

      I am looking for any resource that could tell me how much knowledge and training is needed to go from a beginner to expert, in let's say application software development for example. Or in artificial intelligence.

      If there isn't any one source, are there any general type of sources I can use to piece together one mega-source?

      14 votes
    6. What editor do you use?

      Hey y'all, first time actually posting something here! Just curious what editor people use, whether its for coding, writing, or just the occasional note, whatever. I've gone through most of the...

      Hey y'all, first time actually posting something here! Just curious what editor people use, whether its for coding, writing, or just the occasional note, whatever. I've gone through most of the well-known ones (vim, emacs, atom, vs code for starters), but only ever really messed around with vim enough to like it, and I've also been trying out gedit for the last little while and really liking it, but I'm curious to see what other people use!

      33 votes
    7. XML Data Munging Problem

      Here’s a problem I had to solve at work this week that I enjoyed solving. I think it’s a good programming challenge that will test if you really grok XML. Your input is some XML such as this:...

      Here’s a problem I had to solve at work this week that I enjoyed solving. I think it’s a good programming challenge that will test if you really grok XML.

      Your input is some XML such as this:

      <DOC>
      <TEXT PARTNO="000">
      <TAG ID="3">This</TAG> is <TAG ID="0">some *JUNK* data</TAG> .
      </TEXT>
      <TEXT PARTNO="001">
      *FOO* Sometimes <TAG ID="1">tags in <TAG ID="0">the data</TAG> are nested</TAG> .
      </TEXT>
      <TEXT PARTNO="002">
      In addition to <TAG ID="1">nested tags</TAG> , sometimes there is also <TAG ID="2">junk</TAG> we need to ignore .
      </TEXT>
      <TEXT PARTNO="003">*BAR*-1
      <TAG ID="2">Junk</TAG> is marked by uppercase characters between asterisks and can also optionally be followed by a dash and then one or more digits . *JUNK*-123
      </TEXT>
      <TEXT PARTNO="004">
      Note that <TAG ID="4">*this*</TAG> is just emphasized . It's not <TAG ID="2">junk</TAG> !
      </TEXT>
      </DOC>
      

      The above XML has so-called in-line textual annotations because the XML <TAG> elements are embedded within the document text itself.

      Your goal is to convert the in-line XML annotations to so-called stand-off annotations where the text is separated from the annotations and the annotations refer to the text via slicing into the text as a character array with starting and ending character offsets. While in-line annotations are more human-readable, stand-off annotations are equally machine-readable, and stand-off annotations can be modified without changing the document content itself (the text is immutable).

      The challenge, then, is to convert to a stand-off JSON format that includes the plain-text of the document and the XML tag annotations grouped by their tag element IDs. In order to preserve the annotation information from the original XML, you must keep track of each <TAG>’s starting and ending character offset within the plain-text of the document. The plain-text is defined as the character data in the XML document ignoring any junk. We’ll define junk as one or more uppercase ASCII characters [A-Z]+ between two *, and optionally a trailing dash - followed by any number of digits [0-9]+.

      Here is the desired JSON output for the above example to test your solution:

      {
        "data": "\nThis is some data .\n\n\nSometimes tags in the data are nested .\n\n\nIn addition to nested tags , sometimes there is also junk we need to ignore .\n\nJunk is marked by uppercase characters between asterisks and can also optionally be followed by a dash and then one or more digits . \n\nNote that *this* is just emphasized . It's not junk !\n\n",
        "entities": [
          {
            "id": 0,
            "mentions": [
              {
                "start": 9,
                "end": 18,
                "id": 0,
                "text": "some data"
              },
              {
                "start": 41,
                "end": 49,
                "id": 0,
                "text": "the data"
              }
            ]
          },
          {
            "id": 1,
            "mentions": [
              {
                "start": 33,
                "end": 60,
                "id": 1,
                "text": "tags in the data are nested"
              },
              {
                "start": 80,
                "end": 91,
                "id": 1,
                "text": "nested tags"
              }
            ]
          },
          {
            "id": 2,
            "mentions": [
              {
                "start": 118,
                "end": 122,
                "id": 2,
                "text": "junk"
              },
              {
                "start": 144,
                "end": 148,
                "id": 2,
                "text": "Junk"
              },
              {
                "start": 326,
                "end": 330,
                "id": 2,
                "text": "junk"
              }
            ]
          },
          {
            "id": 3,
            "mentions": [
              {
                "start": 1,
                "end": 5,
                "id": 3,
                "text": "This"
              }
            ]
          },
          {
            "id": 4,
            "mentions": [
              {
                "start": 289,
                "end": 295,
                "id": 4,
                "text": "*this*"
              }
            ]
          }
        ]
      }
      

      Python 3 solution here.

      If you need a hint, see if you can find an event-based XML parser (or if you’re feeling really motivated, write your own).

      4 votes
    8. In search of the dark mode holy grail

      I've been thinking a lot about dark mode lately, now that macOS and Windows 10 both officially offer some implementation of it. I think dark modes make a compelling case for eye strain prevention,...

      I've been thinking a lot about dark mode lately, now that macOS and Windows 10 both officially offer some implementation of it. I think dark modes make a compelling case for eye strain prevention, but the dealbreaker for me is revealed when switching between apps and one of them isn't dark. That jarring flash of bright light completely ruins whatever gentleness the dark environment provided in the first place. So despite my curiosity I've kept everything in light mode for years, tempered by f.lux to keep myself sane after sundown.

      Anyway, now that there's official OS support I'm reconsidering. I think there's a growing pro-dark movement that was just waiting for that formal recognition. Today the programs I use most all offer dark modes so I'm taking an experimental plunge. My goal: 90% elimination of white flashes while in my normal workflow.

      The biggest obstacle is, not surprisingly, the web. There are some beautiful dark browser themes available but that really only affects the UI elements around the page, not the page itself. I want to darken the web too. I have a few thoughts about this:

      • Plugins like this one try to automate a dark mode for every site you visit. This is hit-or-miss, resulting in ugly color combinations, sometimes unreadable text. Some methods just invert the page colors, which can lead to all sort of other visual wonkiness. I haven't found a plugin like this that isn't fiddly and annoying.
      • This plugin looks interesting. From what I can tell, it uses some kind of server-side heuristics to determine the optimal way to darken every page you visit. I haven't actually tried it because I'm concerned about the privacy/security implications of sending all my web activity to this unknown third party. Or what kind of performance hit that would involve. Also, they bury this information on their site, but this is a paid service with an annual subscription.
      • I'm aware of Stylish and its huge library of user-maintained custom site styles. This seemed like a good approach, except that following a recent acquisition, the new owners of Stylish betrayed their users' trust in a very shady way so I'm afraid to go near it now. If there's a credible alternative with a decent style library I'd love to know about it. Especially if there's a way to automate style application so I don't have to manually activate it for every site I visit.
      • Tangentially, the W3C is having an interesting conversation about adding CSS media query support for recognizing user dark-mode preferences. This could absolutely be the future of the web(!!), but I suspect it won't because it puts the responsibility on designers to basically double the amount of work they have to do. Speaking as someone in that field, I would not want to have to add this to my already-long list of design considerations.

      Are there any other good web darkening methods I've overlooked? How do you deal with the white flash problem? Should I just give up and go back to black-on-white? Interested in any and all thoughts on the matter.

      24 votes
    9. Programming Challenge: Polygon analysis.

      It's time for another programming challenge! Given a list of coordinate pairs on a 2D plane that describe the vertices of a polygon, determine whether the polygon is concave or convex. Since a...

      It's time for another programming challenge!

      Given a list of coordinate pairs on a 2D plane that describe the vertices of a polygon, determine whether the polygon is concave or convex.

      Since a polygon could potentially be any shape if we don't specify which vertices connect to which, we'll assume that the coordinates are given in strict order such that adjacent coordinates in the list are connected. Specifically, if we call the list V[1, n] and say that V[i] <-> V[j] means "vertex i and vertex j are connected", then for each arbitrary V[i] we have V[i-1] <-> V[i] <-> V[i+1]. Moreover, since V[1] and V[n] are at the ends of the list, V[1] <-> V[n] holds (i.e. the list "wraps around").

      Finally, for simplicity we can assume that all coordinates are unique, that all polygon descriptions generate valid polygons with 3 or more non-overlapping sides, and that, yes, we're working with coordinates that exist in the set of real numbers only. Don't over-complicate it :)

      For those who want an even greater challenge, extend this out to work with 3D space!

      8 votes
    10. Programming Challenge: Reverse Polish Notation Calculator

      It's been nearly a week, so it's time for another programming challenge! This time, let's create a calculator that accepts reverse Polish notation (RPN), also known as postfix notation. For a bit...

      It's been nearly a week, so it's time for another programming challenge!

      This time, let's create a calculator that accepts reverse Polish notation (RPN), also known as postfix notation.

      For a bit of background, RPN is where you take your two operands in an expression and place the operator after them. For example, the expression 3 + 5 would be written as 3 5 +. A more complicated expression like (5 - 3) x 8 would be written as 5 3 - 8 x, or 8 5 3 - x.

      All your program has to do is accept a valid RPN string and apply the operations in the correct order to produce the expected result.

      18 votes
    11. Programming Challenge: Counting isolated regions.

      Another week, another challenge! This time, assume you're given a grid where each . represents an empty space and each # represents a "wall". We'll call any contiguous space of .s a "region". You...

      Another week, another challenge!

      This time, assume you're given a grid where each . represents an empty space and each # represents a "wall". We'll call any contiguous space of .s a "region". You can also think of a grid with no walls the "base" region. The walls may subdivide the base region into any number of isolated sub-regions of any shape or size.

      Write a program that will, given a grid description, compute the total number of isolated regions.

      For example, the following grid has 5 isolated regions:

      ....#....#
      ....#.###.
      ....#.#.#.
      #...#..#..
      .#..#...#.
      
      16 votes
    12. Batch-saving websites for offline viewing

      Anybody here have a good setup for batch-downloading articles/news from several sites you specify, similar to youtube-dl but for general websites? I'm sure it could be scripted with not too much...

      Anybody here have a good setup for batch-downloading articles/news from several sites you specify, similar to youtube-dl but for general websites? I'm sure it could be scripted with not too much effort but I'm interested what polished solutions there are.

      The idea would be so people with rare internet access could go to a hotspot weekly or something and sync that week's worth of content.

      12 votes
    13. Programming Challenge: Merge an arbitrary number of arrays in sorted order.

      It looks like it's been over a week and a half since our last coding challenge, so let's get one going. This challenge is a relatively simple one, but it's complex enough that you can take a...

      It looks like it's been over a week and a half since our last coding challenge, so let's get one going. This challenge is a relatively simple one, but it's complex enough that you can take a variety of different approaches to it.

      As the title suggests, write a program that accepts an arbitrary number of arrays, in whatever form or manner you see fit (if you want to e.g. parse a potentially massive CSV file, then go nuts!), and returns a single array containing all of the elements of the other arrays in sorted order. That's it!

      Bonus points for creative, efficient, or generalized solutions!

      24 votes
    14. Programming Challenge: Compute the shortest path to visit all target spots on a grid.

      Let's do something a little more challenging this time. Given an MxN grid of arbitrary size, and given a random starting place on that grid and a list of points to visit, find the shortest path...

      Let's do something a little more challenging this time.

      Given an MxN grid of arbitrary size, and given a random starting place on that grid and a list of points to visit, find the shortest path such that you visit all of them. Path lengths will be computed using taxicab distances rather than strict coordinate distance calculations.

      There are no restrictions on expected input this time. Output should be the total distance traveled between points.


      Example

      Assume that we use the character # to denote a spot on the grid, the character @ to denote your starting point, and the character * to denote a place on the grid that you're required to visit. One such grid may look something like this:

      ######
      ######
      **####
      #*####
      #*#*##
      #@####
      ######
      

      In this case, let's say that the bottom-left point on the grid is point (0, 0) and we're starting on point (1, 1). One valid solution would be to move to point (3, 2), then (1, 2), then (1, 3), then (1, 4), and finally (0, 4). The shortest path available is thus 8. Note that it's not enough just to visit the next nearest point on the grid!

      15 votes
    15. An informal look at the concept of reduction (alternatively: problem-solving for beginners).

      Preface One of the most common questions I see from prospective programmers and computer scientists is "where should I start?". My answer to that is a pretty consistent one: learn how to solve...

      Preface

      One of the most common questions I see from prospective programmers and computer scientists is "where should I start?". My answer to that is a pretty consistent one: learn how to solve problems effectively. But that's vague and not really all that helpful, so I figured that I should actually tackle this in a little more depth by touching on something more specific.

      Specifically, I want to touch on the subject of how to think about complex problems.


      The Rationale Behind Learning

      Before we can better understand how to effectively solve problems, it's important to consider how it is that we learn. With any subject, the standard approach is to begin with the bare basics. For programming, that's writing a Hello, World! program in the new language you're working with. For foreign languages, you learn basic common words and sentence structure. For math, you learn your basic arithmetic operations like addition and multiplication.

      From there, we add on more additional complexity and string together everything we've learned. For a foreign language, this looks like learning about new words, stringing them together in your own sentences, then learning about verb tenses and throwing them into the mix as well. With math, you take your normal number crunching and suddenly throw the concept of order of operations into the mix, then variables and how to solve for them.

      As a general rule, we first get comfortable with solving a simple problem and gradually build up toward solving increasingly more difficult ones.


      The Missing Piece

      Odds are that we've all sat in a math class at one point, and when the teacher asked a student how to solve a problem, they received an immediate "I don't know". You may or may not have been that kid yourself. I have no intention of shaming the kids who struggled (or those who still struggle) with math. Rather, I want to point to what I believe is the fundamental cause of that mental barrier that has frustrated students for generations.

      Learning is not simply a matter of adding more complexity to problems. A key part of learning, and one that I don't recall ever having emphasized during my grade school studies, is your ability to break problems down into the steps that you know how to complete and combine the different, simpler skills you've already learned to arrive at a solution. Instead, you were expected to solve many of those complex problems and learn through practice, or through pure rote memorization.

      What determined whether or not you could solve those problems was then a question of whether or not you could intuit or memorize how to solve those specific problems, and brand new problems that still made use of the same skill sets but had completely different forms would throw a wrench in that. Those who could solve any of those problems--those who, I would argue, were often mistakenly referred to as "geniuses" or "talented"--were really just those who knew how to break a problem down into simpler pieces.

      This isn't a failing on the students, but on the way they've been taught to think about problems.


      Reducing Problems

      What does it mean to "break down" a problem, though? The few times I recall a teacher ever touching on the subject, "break down the problem" and "use the skills you've already learned" were the kinds of pieces of advice passed around, completely vague and devoid of meaning for anyone who didn't already understand. How can we better grasp this important step?

      There's a term in complexity theory known as "reduction". The general idea is that if you have problems A and B, where you already know how to solve B, then if you can transform problem A so that it looks like problem B, then you can use your solution for B to solve at least part of A.

      In other words, finding the solution to a more complex problem is just a matter of finding a way to make it look like a problem you already know how to solve.

      The advice to "break down" a problem really means to perform this process of "reduction", of transforming your more complicated problem A into your simpler, known problem B.


      In Practice

      We're still discussing a vague concept, but now that we have more specific language to work with, we can more easily see how it works in practice (a reduction of its own!).

      Let's consider a conceptually simple problem: grabbing the kth largest (or smallest) item from a list. How do we solve this problem? Probably the most obvious and straightforward answer is to sort the list then grab the kth item, right?

      Notice that we gave two high-level descriptions of the steps we need to solve this problem: sorting, then grabbing the appropriate item. We can therefore then state that the problem of "grab the kth largest/smallest item from a list" can be reduced to the two problems "sort a list" and "grab the kth item from a list".

      Now, let's say we're given the problem "take this list of competitor times from the race and tell me what the top 10 race times were". What do we know about this problem? We know that we're being given a list, and we know that we need the 10 smallest items from that list. We also know that "10 smallest items" is just shorthand for "the 1st smallest item, the 2nd smallest item, ..., and the 10th smallest item". We can therefore reduce this problem to the previous one we solved by transforming it into "grab the kth smallest item from a list" and "repeat for values 1-10 for k".


      Practical Advice

      In the end, my explanation may not have helped much at all in actually grasping the concept of reduction. My intent isn't necessarily to help you understand it immediately, but to provide you a framework for a way of thinking. Even if you do grasp the general concept, you may even wonder how you're supposed to recognize these kinds of reductions out in the wild in non-academic environments. The answer, perhaps annoying, is practice. Much like an appraiser can only become good at discerning details through experience, a programmer or computer scientist can only recognize these patterns through repeated exposure.

      In general, if I had to narrow it down to a small list of tips for improving your problem solving skills, this would be it:

      • Work on grasping the concept of reduction itself.
      • Expose yourself to lots of new problems.
      • Don't shy away from difficult problems. Reduce them as much as you can and solve the pieces you're able to. Try to research the pieces you're struggling with. Return to the problem later when you have more experience if you have to, but take a crack at it first.
      • Don't accept "I don't know" as an answer in itself. Ask yourself why you don't how to solve a problem. Narrow down which pieces you're able to solve and which pieces you're not.
      • Just solve problems. Any problems. Easy ones, hard ones, and anything in between. Solving problems is a skill, and practicing it will make you better at solving problems in general, and better at recognizing the simpler problems inside of more complicated ones.
      • Don't just come up with a solution to a problem. Ensure that you understand how each piece of it works and why it works. Copy-pasting from StackOverflow can be a valid tool at your disposal, but doing so mindlessly isn't nearly as valuable as reviewing the solution, being able to determine whether or not it works before ever executing the code, and being able to discard anything unnecessary from it.

      Final Thoughts

      I'm not an authoritative voice on this subject. I'm not an educator. More than anything, I'm a life-long student and an enthusiast. There's seldom a day when I don't have to research something new in order to solve a problem I'm not familiar with, or remind myself the syntax for a function I've used several times in the past. I don't know anything about teaching others, but I do know plenty about learning, and if there's anything that has stood out to me over the years, it's the fact that I find it easier to learn about something or to solve a problem if I can transform the concept into something that's easier for me to grasp.

      Moreover, I'm human and thus prone to mistakes. Call me out on them if you notice them. I'll take any of my mistakes as learning opportunities :)

      11 votes