• Activity
  • Votes
  • Comments
  • New
  • All activity
    1. Light Analysis of a Recent Code Refactor

      Preface In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer...

      Preface

      In a previous topic, I'd covered the subject of a few small lessons regarding code quality. Especially important was the impact on technical debt, which can bog down developer productivity, and the need to pay down on that debt. Today I would like to touch on a practical example that I'd encountered in a production environment.


      Background

      Before we can discuss the refactor itself, it's important to be on the same page regarding the technologies being used. In my case, I work with PHP utilizing a proprietary back-end framework and MongoDB as our database.

      PHP is a server-side scripting language. Like many scripting languages, it's loosely typed. This has some benefits and drawbacks.

      MongoDB is a document-oriented database. By default it's schema-less, allowing you to make any changes at will without an update to schema. This can blend pretty well with the loose typing of PHP. Each document is represented using a JSON-like structure and is stored in something called a "collection". For those of you accustomed to using relational database, a "collection" is analogous to a table, each document is a row, and each field in the document is a column. A typical query in the MongoDB shell would look something like this:

      db.users.findOne({
          username: "Emerald_Knight"
      });
      

      The framework itself has some framework-specific objects that are held in global references. This makes them easily accessible, but naturally littering your code with a bunch of globals is both error-prone and an eyesore.


      Unexpected Spaghetti

      In my code base are a number of different objects that are designed to handle basic CRUD-like operations on their associated database entries. Some of these objects hold references to other objects, so naturally there is some data validation that occurs to ensure that the references are both valid and authorized. Pretty typical stuff.

      What I noticed, however, is that the collection names for these database entries were littered throughout my code. This isn't necessarily a bad thing, except there were some use cases that came to mind: what if it turned out that my naming for one or more of these collections wasn't ideal? What if I wanted to change a collection name for the sake of easier management on the database end? What if I have a tendency to forget the name of a database collection and constantly have to look it up? What if I make a typo of all things? On top of that, the framework's database object was stored in a global variable.

      These seemingly minor sources of technical debt end up adding up over time and could cause some serious problems in the worst case. I've had breaking bugs make their way passed QA in the past, after all.


      Exchanging Spaghetti for Some Light Lasagna

      The problem could be characterized simply: there were scoping problems and too many references to what were essentially magic strings. The solution, then, was to move the database object reference from global to local scope within the application code and to eliminate the problem of magic strings. Additionally, it's a good idea to avoid polluting the namespace with an over-reliance on constants, and using those constants for database calls can also become unsightly and difficult to follow as those constants could end up being generally disconnected from the objects they're associated with.

      There turned out to be a nice, object-oriented, very PHP-like solution to this problem: a so-called "magic method" named "__call". This method is invoked whenever an "inaccessible" method is called on the object. Using this method, a database command executed on a non-database object could pass the command to the database object itself. If this logic were placed within an abstract class, the collection could then be specified simply as a configuration option in the inheriting class.

      This is what such a solution could look like:

      <?php
      
      abstract class MyBaseObject {
      
          protected $db = null;
          protected $collection_name = null;
      
          public function __construct() {
              global $db;
              
              $this->db = $db;
          }
      
          public function __call($method_name, $args) {
              if(method_exists($this->db, $method_name)) {
                  return $this->executeDatabaseCommand($method_name, $args);
              }
      
              throw new Exception(__CLASS__ . ': Method "' . $method_name . '" does not exist.');
          }
      
          public function executeDatabaseCommand($command, $args) {
              $collection = $this->collection_name;
              $db_collection = $this->db->$collection;
      
              return call_user_func_array(array($db_collection, $command), $args);
          }
      }
      
      class UserManager extends MyBaseObject {
          protected $collection_name = 'users';
      
          public function __construct() {
              parent::__construct();
          }
      }
      
      $user_manager = new UserManager();
      $my_user = $user_manager->findOne(array('username'=>'Emerald_Knight'));
      
      ?>
      

      This solution utilizes a single parent object which transforms a global database object reference into a local one, eliminating the scope issue. The collection name is specified as a class property of the inheriting object and only used in a single place in the parent object, eliminating the magic string and namespace polluting issues. Any time you perform queries on users, you do so by using the UserManager class, which guarantees that you will always know that your queries are being performed on the objects that you intend. And finally, if the collection name for an object class ever needs to be updated, it's a simple matter of modifying the single instance of the class property $collection_name, rather than tracking down some disconnected constant.


      Limitations

      This, of course, doesn't solve all of the existing problems. After all, executing the database queries for one object directly from another is still pretty bad practice, violating the principle of separation of concerns. Instead, those queries should generally be encapsulated within object methods and the objects themselves given primary responsibility in handling associated data. It's also incredibly easy to inadvertently override a database method, e.g. defining a findOne() method on UserManager, so there's still some mindfulness required on the part of the programmer.

      Still, given the previous alternative, this is a pretty major improvement, especially for an initial refactor.


      Final Thoughts

      As always, technical debt is both necessary and inevitable. After all, in exchange for not taking the excess time and considering structuring my code this way in the beginning, I had greater initial velocity to get the project off of the ground. What's important is continually reviewing your code as you're building on top of it so that you can identify bottlenecks as they begin to strain your efficiency, and getting those bottlenecks out of the way.

      In other words, even though technical debt is often necessary and is certainly inevitable, it's important to pay down on some of that debt once it starts getting expensive!

      7 votes
    2. Let's talk best-practice Jenkins on AWS ECS

      [seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly] I've had some...

      [seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly]

      I've had some experience in this realm for a while now, but I'm having a little trouble with one issue in particular. Before I divulge, I'll present my thoughts on best practice and and what I've been able to implement:

      • Terraform everything (in accordance to terragrunt's "style guide" i.e. organization)
        THIS IS A BIG ONE: for the jenkins master task, make sure to use the following args to make sure jenkins jobs aren't super slow as hell to start:
      -Djava.awt.headless=true -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
      

      THIS IS A GAME CHANGER (more-so on k8s clusters when the ecs plugin isn't used... hint, it's shit).

      • Create an EFS (in a separate terraform module) and mount it to the jenkins ECS cluster at /var/jenkins_home. Makes jenkins much more reliable through outages and easier to upgrade.
      • Run a logging agent (via docker container) like logspout or newrelic or whatever IN USER_DATA and not as a task - that way you get logs if there are issues during user_data/cloud_init... this I'm actually not sure about. Running a container outside the context of an ECS task means the ECS agent can't really track it and allocate mem/cpu properly... but it does help with user_data triage.
      • Use pipelines and git plugins to drive jobs. All jenkins jobs should be in source control!
      • Make sure you setup docker cleanup jobs on DAY 1! If you hace limited access to your cluster and you run out of disk due to docker cache, networks, volumes, etc... you're screwed till the admin ssh's in and runs a prune. Get a docker system prune going or the equivalent for each docker resource with appropriate filters... i.e. filter for anything older than a few days and is dangling.
      • Use Jenkins Global Libraries to make Jenkinsfiles cleaner (I always just use vars instead of groovy/java style packages because it's easier and less ugly)
        Jenkinsfiles should mostly call other bash files, make files, python scripts to generate and load prop files, etc. The less logic you put in a Jenkinsfile (which is just modified groovy) the better. String interpolation, among other things, is a fuckery that we don't have time to triage.
      • (out-of-scope) Move to using k8s/EKS instead of ECS asap because the ECS plugin for jenkins is absolute shit and it doesn't use priority correctly (sorry whoever developed it and... oh wait abandoned it and hasn't merged anything for years... for for real it's cool, just give admin to someone else).
      • (cultural) Stop calling them slaves. "Hey @eng, we're rotating slaves due to some cache issues. If you have been affected by race conditions in that past, our new update and slave rotation should fix that. Our update may have killed your job that was running on an old slave, just wait a few and the new slaves will be ready" <--This just doesn't look good.
        Hope that was some good stuff for you guys. Maybe I'm preaching to the choir, but I've seen some pretty shit jenkins setups.

      NOW FOR MY QUESTION!

      Has ANYONE actually been able to setup a proper jenkins user on ECS that actually works for both a master and ephemeral jenkins-agents so that they can mount and use the docker.sock for builds without hitting permission issues? I'm talking using the ecs plugin and mounting docker.sock via that.

      I have always resorted to running jenkins master and agents as root, which means you have to chmod files (super expensive time and cpu for services with tons of files). Running microservices as root is obviously bad practice, and chmod-ing a zilliion files is shit for docker cache and time... so I want to get jenkins users able to utilize the docker.sock. THIS IS SPECIFICALLY FOR THE AWS ECS AMI! I don't care about debian or old versions of docker where you could use DOCKER_OPTS. That doesn't work on the AWS Linux image.

      Thanks! And happy Friday!

      5 votes
    3. any sites like 23andme that dont sell your DNA to corporations?

      I'd really like to have a DNA test done to know my family history beyond 2 generations (adopted relatives) I've heard numerous times that 23andme will abuse the information they obtain and either...

      I'd really like to have a DNA test done to know my family history beyond 2 generations (adopted relatives)

      I've heard numerous times that 23andme will abuse the information they obtain and either target you with ads or sell your DNA to marketing agencies, are there any non invasive DNA tests available online or elsewhere?

      14 votes
    4. Good use for Twitter?

      I hear a lot about how Twitter is a bad concept, because the character limit means things can be oversimplified, taken out of context, and posted without a source. I've never used it myself, but...

      I hear a lot about how Twitter is a bad concept, because the character limit means things can be oversimplified, taken out of context, and posted without a source. I've never used it myself, but recently I've been wondering if it can be useful to me. For example, some bloggers I follow might post some insights on Twitter that they don't anywhere else, and help me discover other interesting blogs.

      Have you found a good use for Twitter besides as a social network or news aggregator?

      13 votes
    5. Black Mirror S04E03 “Crocodile” Discussion Thread

      Previous episode | Index thread | Next episode Black Mirror Season 4 Episode 3 - Crocodile Architect Mia scrambles to keep a dark secret under wraps, while insurance investigator Shazia harvests...

      Previous episode | Index thread | Next episode

      Black Mirror Season 4 Episode 3 - Crocodile

      Architect Mia scrambles to keep a dark secret under wraps, while insurance investigator Shazia harvests people's memories of a nearby accident scene.

      Black Mirror Netflix link


      Warning: this thread contains spoilers about this episode! If you haven't seen it yet, please watch it and come back to this thread later.

      You can talk about past episodes, but please don't discuss future episodes in this thread!


      If you don't know what to say, here are some questions to get the discussion started:

      • How does the title relate to the episode itself?
      • Are there any similarities between real life events and the episode?
      • Are there any references or easter eggs in the episode, such as references to past episodes?

      Please rate the episode here!

      4 votes
    6. Prompted by a recent post, I asked myself: is collecting digital media really considered hoarding?

      From this Tildes post https://tildes.net/~music/7vc/anyone_still_listening_to_music_with_files_instead_of_streaming I started thinking. To me, curating my own collections so that I can experience...

      From this Tildes post https://tildes.net/~music/7vc/anyone_still_listening_to_music_with_files_instead_of_streaming

      I started thinking. To me, curating my own collections so that I can experience them only makes common sense. And I also journal on what I read, watch, and listen to. I've known a few hoarders in my life, who enlarge their homes to house tons of material things that they'll never use and can't throw away.

      In my own case, I'm slowly winnowing down the hundred or so CD's, books, and movies I own just because they take up space and I don't want my kids having to have a massive garage sale when I die. And I really like that I can have a whole library basically on an old phone that holds an SDHC card.

      So is obsession/compulsion with digital media the same as physical hoarding? Is it just as harmful? And how do I class the tons of emails, mostly work, that I don't bother to throw away because it's just too time consuming? The same thing goes for family and self snapped photos. To me they're in a different category altogether.

      Am I the biggest, most hypocritical minimalist ever? Is there such a thing as non-material materialism? What are your justifications for or against streaming vs. accumulating?

      19 votes
    7. What are you reading these days? #5 (Was: What are you reading this week?)

      What are you reading currently? Fiction or non-fiction, any genre, any language! Tell us what you're reading, and talk a bit about it. Notes: I've modified the title a bit, having it say "this...

      What are you reading currently? Fiction or non-fiction, any genre, any language! Tell us what you're reading, and talk a bit about it.

      Notes: I've modified the title a bit, having it say "this week" when it was never weekly (it's bi-weekly) was a bit weird.

      Past weeks: Week #1 · Week #2 · Week #3 · Week #4

      18 votes
    8. Getting Started as a Developer from Scratch

      I have been interested in making the gradual career change to software development from my current humanities field. This stems from a handful of different places. Of course the pay and...

      I have been interested in making the gradual career change to software development from my current humanities field. This stems from a handful of different places. Of course the pay and flexibility are strong drivers but I like the idea of a field that is somewhat of a creative expression; one where you can manifest your knowledge and experience into something tangible.

      I have no experience with programming other than SQL use in ArcGIS and am hoping to gain some knowledge about the field; so anything would be helpful. Whether what to expect from this line of work, where someone with no experience should look to get started and what to expect, personal journeys, etc.

      Cheers!

      14 votes
    9. Same-sex penguin couple at Sea Life Sydney Aquarium become parents

      ABC: Same-sex penguin couple at Sea Life Sydney Aquarium become parents SBS: Sydney's very own gay penguins have welcomed their first baby Sydney Morning Herald: Love birds: Sydney's same-sex...

      This is a follow-up to this previous topic posted here a couple of weeks ago: Same-Sex Penguin Couple Fosters An Egg In Sydney

      12 votes
    10. What does the online / social media world look like to you, what would you want?

      Some of you may have heard that Google+ will be shutting down in August, 2019. Though much criticised (including by me), the site offered some compelling dynamics, and I've reflected a lot on...

      Some of you may have heard that Google+ will be shutting down in August, 2019. Though much criticised (including by me), the site offered some compelling dynamics, and I've reflected a lot on those.

      I'm involved in the effort to find new homes for Plussers and Communities, which has become something of an excuse to explore and redefine what "online" and "social" media are ("PlexodusWiki").

      Part of this involves some frankly embarrassing attempts to try to define what social media is, and what its properties are (both topics reflected heavily in the recent-changes section of the wiki above).

      Tildes is ... among the potential target sites (there are a few Plussers, some of whom I really appreciated knowing and hearing from there), here, though the site dynamics make discovering and following them hard. This site is evolving its own culture and dynamics, parts of which I'm becoming aware of.

      I've been online for well over 30 years, and discovered my first online communities via Unix talk, email, FTP, and Usenet, as well as (no kidding) a computerised university library catalogue system. Unsurprisingly: if you provide a way, especially for bright and precocious minds to interact with one another, they will. I've watched several evolutions of Internet and Web, now increasing App-based platforms. There are differences, but also similarities and patterns emerging. Lessons from previous eras of television, radio, telephony, telegraphy, print, writing, oral traditions, and more, can be applied.

      I've got far more questions than answers and thought I'd put a few out here:

      • What does online or social media mean to you? Is it all user-generated content platforms? Web only? Apps? Email or chat? Wikis? GitHub, GitLab, and StackExchange?

      • Is social networking as exemplified by Facebook or Twitter net good or bad? Why? If bad, how might you fix it? Or is it time to simply retreat?

      • What properties or characteristics would you use to specify, define, or distinguish social or online media?

      • What emergent properties -- site dynamics, if you will -- are positive or negative? What are those based on?

      • What are the positive and negative aspects of scale?

      • What risks would you consider in self-hosting either your own or a group's online presence?

      • What is/was the best online community experience you've had? What characterised it? How did it form? How did it fail (if it did)?

      • What elements would comprise your ideal online experience?

      • What would you nuke from orbit, after takeoff, just to be sure?

      • Are you or your group seeking new options or platforms? What process / considerations do you have?

      I could keep going and will regret not adding other questions, but this is a good start. Feel free to suggest other dimensions, though some focus on what I've prompted with would be appreciated.

      19 votes
    11. Does anyone here share a passion for spiritual development, the occult, metaphysics, or fringe science/academia?

      One of my biggest hobbies and passions over the last 10 or 15 years has been essentially all of the above. I'm not the smartest or the most well-read lady out there by any means but I enjoy...

      One of my biggest hobbies and passions over the last 10 or 15 years has been essentially all of the above. I'm not the smartest or the most well-read lady out there by any means but I enjoy exploring the more shadowy realms of discourse. There's lots and lots of dross but occasionally a nugget of something magnificent, and over the years it's eroded away my original scientific materialist atheism completely and my thinking now is more animist, panpsychist, deist. I've spent years off and on experimenting with (actual, not stage) magic, and though I was never super committed to the full ceremonial experience like others I've seen, it's become a part of how I think.

      So I was wondering if there's any here that don't fit into the typical scientific materialist box in one form or another. And if so, what're you reading or experimenting with right now?

      Currently I'm reading through Conversations with God and it's persuaded me to start practicing loving-kindness meditation. I've only been at that a few days but I'm interested to see what impact it has on my daily life. It's definitely true that up until these past few days I've never actively focused on trying to love myself and others, which kind of surprises me when I think about it. But that sort of thing isn't really something I see emphasized in our culture or in my own little circle.

      How about you?

      21 votes
    12. Who do you think the archtypical 'bad guys' might be in film and video games in the future?

      Obviously this depends heavily on one's interpretation and subsequent extrapolation of the current geopolitical climate. That being said Nazis, the Soviet Union and generic Middle Eastern...

      Obviously this depends heavily on one's interpretation and subsequent extrapolation of the current geopolitical climate. That being said Nazis, the Soviet Union and generic Middle Eastern terrorists will eventually fall out of vogue and lose their cultural significance with each passing generation. What might come next?

      If nothing else, I suspect zombies will remain one of those recurring trends that come in waves.

      16 votes
    13. What are the best practices for passphrase security?

      This is a sort of continuation of a previous topic I posted. This weekend I will be wiping and reinstalling my computer and encrypting all of my drives in the process. In doing so, I will have to...

      This is a sort of continuation of a previous topic I posted. This weekend I will be wiping and reinstalling my computer and encrypting all of my drives in the process. In doing so, I will have to choose secure passphrases. As such, I have some questions about how best to do this:

      1. I have three drives that will be encrypted. Is it okay to have the same passphrase for all of them, or should I have different ones for each?

      2. In looking up info on this topic, I came across this article which recommends something called a Diceware wordlist. The premise is that you roll dice which match to a list of 7000+ words. You then string six or more of these words together which become your passphrase. Is this a sound way to generate one?

      3. Rather than using the Diceware wordlist, couldn't I roll my own password of the same type using six "random" words of my choosing? I feel like that would be easier to remember, but am I weakening security in doing so?

      4. If the Diceware method is to be trusted, does that mean I do not need to pepper my passphrase with digits, mixed case, and special characters? Or should I add these anyway?

      5. I'm also considering changing over passwords on a lot of my online accounts based on this method. I like the idea of using a single passphrase as a root, but how do you modify it so that it is different for each account? Would I do something like [dicewarewords]tildes, [dicewarewords]spotify, [dicewarewords]ubuntuforums, etc.? I feel like it would be too on-the-nose, and it would make it easy to guess my other passwords if one were compromised. On the other hand, I don't like the idea of using a password manager to generate a random string for me. I'd like to still be able to login even without my password manager.

      6. For people that have used something like this, how do you then deal with password restrictions on sites? I know that no matter how great I set things up I'm still going to have to make exceptions for sites that that either require or forbid numbers, mixed case, or special characters, have character limits, or make me change my password frequently.

      14 votes
    14. Tildes can now receive Basic Attention Tokens (from the Brave browser or BATify extension)

      We've had a few topics related to the Brave browser lately, and at some users' urging, I've now set up and verified Tildes to be able to receive the Basic Attention Token that it allows you to...

      We've had a few topics related to the Brave browser lately, and at some users' urging, I've now set up and verified Tildes to be able to receive the Basic Attention Token that it allows you to allocate to sites that you visit often. Outside of Brave itself, there's also an extension called BATify that seems to allow you to use BAT from Chrome or Firefox.

      I'm not sure if this will ever be a significant source of donations for the site, but it's probably good to have it as an option anyway.

      I haven't tried Brave myself yet, so I can't endorse it personally, but quite a few people seem to like it and it just had a major update last week that made it Chromium-based. If you're thinking about trying it out, I'd appreciate it if you could download it through this referral link:

      https://brave.com/til524

      I don't know the exact details, but it should give Tildes about $5 USD in BAT for each user who "downloads the Brave browser using the promo link specific to your web site and uses the browser (minimally) over a 30 day period".

      I'll add some info about this to the Donate page on the docs site as well, and if anyone that knows more about Brave/BAT than me (which is a very low bar) notices anything wrong or that I should change, please let me know.

      56 votes
    15. ~music Listening Club 19 - The Beatles (The White Album)

      19 weeks and there's another classic record discussion to be had: The Beatles by The Beatles! The Beatles, also known as "The White Album", is the ninth studio album by the English rock band the...

      19 weeks and there's another classic record discussion to be had: The Beatles by The Beatles!

      The Beatles, also known as "The White Album", is the ninth studio album by the English rock band the Beatles, released on 22 November 1968. A double album, its plain white sleeve has no graphics or text other than the band's name embossed, which was intended as a direct contrast to the vivid cover artwork of the band's previous LP Sgt. Pepper's Lonely Hearts Club Band. Although no singles were issued from The Beatles in Britain and the United States, the songs "Hey Jude" and "Revolution" originated from the same recording sessions and were issued on a single in August 1968. The album's songs range in style from British blues and ska to tracks influenced by Chuck Berry and by Karlheinz Stockhausen.

      Here's the place to discuss your thoughts on the record, your history with it or the artist, and basically talk about whatever you want to that goes along with the white album! Remember that this is intended to be a slow moving thing, feel free to take your time and comment at any point in the week!

      If you'd like to stream or buy the album, it can be found on most platforms here.

      Don't forget to nominate and vote for next week's obscure record in response to this comment!

      11 votes