This below is a summary of some real-world performance investigation I recently went through. The tools I used are installed on all linux systems, but I know some people don't know them and would straight up jump to heavyweight log analysis services and what not, or writing their own solution.

Let's say you have request log sampling in a bunch of log files that contain lines like these:

127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000
127.0.0.1 [2021-05-27 23:51:22.019] "GET /pl/player/123456/changelog/ HTTP/1.1" 200 16524 TLSv1.2 MISS-CLUSTER SessionID:(null) Cache:

You might recognize Fastly logs there (IP anonymized). Now, there's a lot you might care about in this log file, but in my case, I wanted to get a breakdown of hits vs misses by URL.

So, first step, let's concatenate all the log files with cat *.log > all.txt, so we can work off a single file.

Then, let's split the file in two: hits and misses. There are a few different values for them, the majority are covered by either HIT-CLUSTER or MISS-CLUSTER. We can do this by just grepping for them like so:

grep HIT-CLUSTER all.txt > hits.txt; grep MISS-CLUSTER all.txt > misses.txt

However, we only care about url and whether it's a hit or a miss. So let's clean up those hits and misses with cut. The way cut works, it takes a delimiter (-d) and cuts the input based on that; you then give it a range of "fields" (-f) that you want.

In our case, if we cut based on spaces, we end up with for example: 127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000.

We care about the 5th value only. So let's do: cut -d" " -f5 to get that. We will also sort the result, because future operations will require us to work on a sorted list of values.

cut -d" " -f5 hits.txt | sort > hits-sorted.txt; cut -d" " -f5 misses.txt | sort > misses-sorted.txt

Now we can start doing some neat stuff. wc (wordcount) is an awesome utility, it lets you count characters, words or lines very easily. wc -l counts lines in an input, since we're operating with one value per line we can easily count our hits and misses already:

$ wc -l hits-sorted.txt misses-sorted.txt
  132523 hits-sorted.txt
  220779 misses-sorted.txt
  353302 total

220779 / 132523 is a 1:1.66 ratio of hits to misses. That's not great…

Alright, now I'm also interested in how many unique URLs are hit versus missed. uniq tool deduplicates immediate sequences, so the input has to be sorted in order to deduplicate our entire file. We already did that. We can now count our urls with uniq < hits-sorted.txt | wc -l; uniq < misses-sorted.txt | wc -l. We get 49778 and 201178, respectively. It's to be expected that most of our cache misses would be in "rarer" urls; this gives us a 1:4 ratio of cached to uncached URL.

Let's say we want to dig down further into which URLs are most often hitting the cache, specifically. We can add -c to uniq in order to get a duplicate count in front of our URLs. To get the top ones at the top, we can then use sort, in reverse sort mode (-r), and it also needs to be numeric sort, not alphabetic (-n). head lets us get the top 10.

$ uniq -c < hits-sorted.txt | sort -nr | head
    815 /static/app/webfonts/fa-solid-900.woff2?d720146f1999
    793 /static/app/images/1.png
    786 /static/app/fonts/nunito-v9-latin-ext_latin-regular.woff2?d720146f1999
    760 /static/CACHE/js/output.cee5c4089626.js
    758 /static/images/crest/3/light/notfound.png
    757 /static/CACHE/css/output.4f2b59394c83.css
    756 /static/app/webfonts/fa-regular-400.woff2?d720146f1999
    754 /static/app/css/images/loading.gif?d720146f1999
    750 /static/app/css/images/prev.png?d720146f1999
    745 /static/app/css/images/next.png?d720146f1999

And same for misses:

$ uniq -c < misses-sorted.txt | sort -nr | head
     56 /
     14 /player/237678/
     13 /players/
     12 /teams/
     11 /players/top/
<snip>

So far this tells us static files are most often hit, and for misses it also tells us… something, but we can't quite track it down yet (and we won't, not in this post). We're not adjusting for how often the page is hit as a whole, this is still just high-level analysis.

One last thing I want to show you! Let's take everything we learned and analyze those URLs by prefix instead. We can cut our URLs again by slash with cut -d"/". If we want the first prefix, we can do -f1-2, or -f1-3 for the first two prefixes. Let's look!

cut -d'/' -f1-2 < hits-sorted.txt | uniq -c | sort -nr | head
 100189 /static
   5948 /es
   3069 /player
   2480 /fr
   2476 /es-mx
   2295 /pt-br
   2094 /tr
   1939 /it
   1692 /ru
   1626 /de

cut -d'/' -f1-2 < misses-sorted.txt | uniq -c | sort -nr | head
  66132 /static
  18578 /es
  17448 /player
  17064 /tr
  11379 /fr
   9624 /pt-br
   8730 /es-mx
   7993 /ru
   7689 /zh-hant
   7441 /it

This gives us hit-miss ratios by prefix. Neat, huh?

Inspired by this post on lobste.rs, I thought it'd be fun for us all to post our homepages and talk about them. I'm posting this in ~creative because I think of a homepage as a creative endeavor, but feel free to move this to ~design or ~tech or wherever, mods.

Just post your homepage as a top-level comment, and we'll workshop in replies!

I'm attempting to provision two mirror staging and production environments for a future SaaS application that we're close to launching as a company, and I'd like to get some feedback on the provisioning script I've created that takes a default VPS from our hosting provider, DigitalOcean, and readies it for being a secure hosting environment for our application instance (which runs inside Docker, and persists data to an unrelated managed database).

I'm sticking with a simple infrastructure architecture at the moment: A single VPS which runs both nginx and the application instance inside a containerised docker service as mentioned earlier. There's no load balancers or server duplication at this point. @Emerald_Knight very kindly provided me in the Tildes Discord with some overall guidance about what to aim for when configuring a server (limit damage as best as possible, limit access when an attack occurs)—so I've tried to be thoughtful and integrate that paradigm where possible (disabling root login, etc).

I’m not a DevOps or sysadmin-oriented person by trade—I stick to programming most of the time—but this role falls to me as the technical person in this business; so the last few days has been a lot of reading and readying. I’ll run through the provisioning flow step by step. Oh, and for reference, Ubuntu 20.04 LTS.

First step is self-explanatory.

#!/bin/sh

# Name of the user to create and grant privileges to.
USERNAME_OF_ACCOUNT=

sudo apt-get -qq update
sudo apt install -qq --yes nginx
sudo systemctl restart nginx

Next, create my sudo user, add them to the groups needed, require a password change on first login, then copy across any provided authorised keys from the root user which you can configure to be seeded to the VPS in the DigitalOcean management console.

useradd --create-home --shell "/bin/bash" --groups sudo,www-data "${USERNAME_OF_ACCOUNT}"
passwd --delete $USERNAME_OF_ACCOUNT
chage --lastday 0 $USERNAME_OF_ACCOUNT

HOME_DIR="$(eval echo ~${USERNAME_OF_ACCOUNT})"
mkdir --parents "${HOME_DIR}/.ssh"
cp /root/.ssh/authorized_keys "${HOME_DIR}/.ssh"

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown --recursive "${USERNAME_OF_ACCOUNT}":"${USERNAME_OF_ACCOUNT}" "${HOME_DIR}/.ssh"

sudo chmod 775 -R /var/www
sudo chown -R $USERNAME_OF_ACCOUNT /var/www
rm -rf /var/www/html

Installation of docker, and run it as a service, ensure the created user is added to the docker group.

sudo apt-get install -qq --yes \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88

sudo add-apt-repository --yes \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

sudo apt-get -qq update
sudo apt install -qq --yes docker-ce docker-ce-cli containerd.io

# Only add a group if it does not exist
sudo getent group docker || sudo groupadd docker
sudo usermod -aG docker $USERNAME_OF_ACCOUNT

# Enable docker
sudo systemctl enable docker

sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version

Disable root logins and any form of password-based authentication by altering sshd_config.

sed -i '/^PermitRootLogin/s/yes/no/' /etc/ssh/sshd_config
sed -i '/^PasswordAuthentication/s/yes/no/' /etc/ssh/sshd_config
sed -i '/^ChallengeResponseAuthentication/s/yes/no/' /etc/ssh/sshd_config

Configure the firewall and fail2ban.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw reload
sudo ufw --force enable && sudo ufw status verbose

sudo apt-get -qq install --yes fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Swapfiles.

sudo fallocate -l 1G /swapfile && ls -lh /swapfile
sudo chmod 0600 /swapfile && ls -lh /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile && sudo swapon --show
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Unattended updates, and restart the ssh daemon.

sudo apt install -qq unattended-upgrades
sudo systemctl restart ssh

Some questions

You can assume these questions are cost-benefit focused, i.e. is it worth my time to investigate this, versus something else that may have better gains given my limited time.

1. Obviously, any critiques of the above provisioning process are appreciated—both on the micro level of criticising particular lines, or zooming out and saying “well why don’t you do this instead…”. I can’t know what I don’t know.

2. Is it worth investigating tools such as ss or lynis (https://github.com/CISOfy/lynis) to perform server auditing? I don’t have to meet any compliance requirements at this point.

3. Do I get any meaningful increase in security by implementing 2FA on login here using google authenticator? As far as I can see, as long as I'm using best practices to actually ssh into our boxes, then the likeliest risk profile for unwanted access probably isn’t via the authentication mechanism I use personally to access my servers.

4. Am I missing anything here? Beyond the provisioning script itself, I adhere to best practices around storing and generating passwords and ssh keys.

Some notes and comments

1. Eventually I'll use the hosting provider's API to spin up and spin down VPS's on the fly via a custom management application, which gives me an opportunity to programmatically execute the provisioning script above and run some over pre- and post-provisioning things, like deployment of the application and so forth.

2. Usage alerts and monitoring is configured within DigitalOcean's console, and alerts are sent to our business' Slack for me to action as needed. Currently, I’m settling on the following alerts:
   1. Server CPU utilisation greater than 80% for 5 minutes.
   2. Server memory usage greater than 80% for 5 minutes.
   3. I’m also looking at setting up daily fail2ban status alerts if needed.

Are you an “old timer” in the computer industry? I’m writing a story about the things programmers (and QA people) had to do to test their software. It’s meant to be a nostalgic piece that’ll remind people about old methods — for good or ill.

For example, there was a point where the only way to insert a breakpoint in the code was to insert “printfs” that said “I got to this place in the code!” And all testing was manual testing. Nothing was automated. If you wanted a bug tracking system, you built your own.

So tell me your stories. Tell me what you had to do to test software, way back when, and compare it to today. What tools did you use -- or build? Is there anything you miss? Anything that makes you especially glad that the past is past?

C’mon, you know you wanted a “remember when”!

This is gonna be kinda of a personal mess.

My background is in film. In Bahia, Brazil.

I understand this is a very personal question with numerous factors to take in, some on which I'll absolutely not be able to convey.

I'm not looking for any definitive life advice because I know that's impossible. I just wanna hear perspectives from some smart people that might help me understand my situation. I've recently been through a (kind of a) life and death situation. I'd be dead or with severe neurological trauma without a helmet.

This made me rethink a lot of stuff about my goals and my life in general. I feel I can confide on Tildes, you people are usually caring and smart and awesome. I'm also a bit emotional, so please be gentle. Spending 24 hours on a hospital bed contemplating death and incapacitation kind does that too with you.

I won't change many details because fuck it, I don't thank there are a lot of people in the world wanting to dox me. And Google already knows everything about me anyway.

I have two very serious psychiatric diagnostics that impart my life in serious ways: bipolar disorder (type II, thankfully) and ADHD. I'm also suspected to be on the autism spectrum but I don't have the means to achieve this diagnostic. It would be useful anyway. These conditions seriously impact my ability to sustain a job for long periods and I have a hard time working with teams bigger than three (sometimes not even than).

I live for free in my mother's conformable apartment, while I she actually spends most of the time on another continent. It's a pretty good deal. But I wanted to be independent.

About two years ago I decided that work in film (my original major) would never provide me the financial independence I needed. Working in film means traveling a lot, infrequent hours, absurd exploration (its common to sleep 4 hours a day), and rampant drug use. I love film and do have a talent for it, but the environment is simply not conducive to my mental health.

Of course, now I realize that computer science may also not be conducive to mental health issues at all. The thing is, really like. When I'm lisping, the real illogical world becames more bearable, and I feel in a wonderland of logic, reason, and calming predictabilidade. This doesn't happen as much with other languages such as Python. I also suck at it. So much that's not even funny. I'm addicted to Linux, Emacs, and the command line, but that's kinda it. I became a Vim/Emacs semi specialist. I don't see myself ever doing anything complex. It this my mind, really!

I've been trying to program for almost 3 years and, beside my super awesome machine, I have nothing to show for myself. I try focusing on using things like Java or Python but I always get sidetracked trying to do some cool shit on Emacs.

Sometimes I wonder if I should just assume that I won't be able to concentrate on anything else and just learn Emacs Lisp for real. It's frowned upon by a lot of people, but Emacs is a wonderful learning environment and at least I would be doing something. Maybe an interesting package that some people would like to use.

Right now my choice seems to be between failing to study things that make me miserable (like OOP), but have clear professional possibilities, or focusing on something I actually like that might make a better programmer in the future.

An important detail: I'm 38 years old and unemployed. My region is not very economically active in that area but I'm afraid to leave it because then I would lose my support network. And the mere notion of being with other people on a daily bases causes me panic attacks.

And, as a reminder, studying programming with bipolar disorder ADHD is hard as fuck. My ADHD is so severe that I constantly forget what I'm doing withing seconds. That's probably why I like Lisp, which is more regular than other languages and I can get things more easily from context.

On the other hand, I'm super charming (and not at all modest hahaha) and interesting at parties because my scattered interests make it possible to contribute meaningfully (and sometimes witty) to pretty much any conversation. My success with women is indirectly proportional to may financial troubles.

Anyway, I know I said this was not about advice, but I kinda lied: what's your advice? Should I keep trying on something I'm not really talented at just because I like it (and it may bring financial rewards in the future).

Or should I just give up and, try my hand at some shorts and even a novel? (I'm currently on a severe writer's block though, but I do have some talent for it).

Maybe I could work from home, be some kind of sysadmin (in which case, what would be the quickest and cheapest way to do so?). I absolutely don't wanna create huge complex products, but managing thinks remotely would be awesome.

I also love philosophy and logic, and, if became suddenly rich, that's what I'd do for the rest of my life. Oh, well.

So recently I've been playing Destiny 2. After you launch the game you need to press "X" for the game to start loading (which takes multiple minutes, it's ridiculous).

I've seen this in other games and I never understood the point. Yes I want to move past that screen, load the game and play it. Do you guys know why game developers do this?

Input wanted for an article.

Let's say that your company is considering the purchase of an expensive new application to help in the company's software development. The demo looks great, and the feature list makes it sound perfect for your needs. So your Management arranges for a proof of concept license to find out if the software is worth the hefty investment. The boss comes to you to ask you to be in charge of the PoC project.

I'm aiming to write an article to help developers, devops, and testers determine if a given vendor's application meets the company's needs. The only assumption I'm making is that the software is expensive; if it's cheap, the easy answer is, "Buy a copy for a small team and see what they think." And I'm thinking in terms of development software rather than enterprise tools (e.g. cloud-based backup) though I suspect many of the practices are similar.

Aside: Note that this project is beyond "Decide if we need such a thing." In this scenario, everyone agrees that purchasing a tool is a good idea, and they agree on the baseline requirements. The issue is whether this is the right software for the job.

So, how do you go about it? I'm sure that it's more than "Get a copy and poke at it randomly." How did (or would) you go about designing a PoC project? If you've been involved in such a project in the past (particularly if the purchase wasn't ideal), what advice could someone have given you to help you make a better choice? I want to create a useful guide that applies to any "enterprise-class" purchase.

For example: Do you recommend that the PoC period be based on time (N months) or workload (N transactions)? How do you decide who should be on the PoC team? What's involved in putting together a comprehensive list of requirements (e.g. integrates with OurFavoredDatabase, meets performance goals of X), creating a test suite that exercises what the software dev product does, and evaluating the results? ...and what am I not thinking of, that I should?