Sometimes a comment thread is very very toxic/controversial and I would like to avoid getting sucked into it. Just because my decision-making is much better at 2pm than it is at 2am. I understand I can and should exercise self-control, and I'm working on it, I assure you! In the meantime, if at all possible, it would be nice to remove certain comments from my view, along with its children. Thanks!

This below is a summary of some real-world performance investigation I recently went through. The tools I used are installed on all linux systems, but I know some people don't know them and would straight up jump to heavyweight log analysis services and what not, or writing their own solution.

Let's say you have request log sampling in a bunch of log files that contain lines like these:

127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000
127.0.0.1 [2021-05-27 23:51:22.019] "GET /pl/player/123456/changelog/ HTTP/1.1" 200 16524 TLSv1.2 MISS-CLUSTER SessionID:(null) Cache:

You might recognize Fastly logs there (IP anonymized). Now, there's a lot you might care about in this log file, but in my case, I wanted to get a breakdown of hits vs misses by URL.

So, first step, let's concatenate all the log files with cat *.log > all.txt, so we can work off a single file.

Then, let's split the file in two: hits and misses. There are a few different values for them, the majority are covered by either HIT-CLUSTER or MISS-CLUSTER. We can do this by just grepping for them like so:

grep HIT-CLUSTER all.txt > hits.txt; grep MISS-CLUSTER all.txt > misses.txt

However, we only care about url and whether it's a hit or a miss. So let's clean up those hits and misses with cut. The way cut works, it takes a delimiter (-d) and cuts the input based on that; you then give it a range of "fields" (-f) that you want.

In our case, if we cut based on spaces, we end up with for example: 127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000.

We care about the 5th value only. So let's do: cut -d" " -f5 to get that. We will also sort the result, because future operations will require us to work on a sorted list of values.

cut -d" " -f5 hits.txt | sort > hits-sorted.txt; cut -d" " -f5 misses.txt | sort > misses-sorted.txt

Now we can start doing some neat stuff. wc (wordcount) is an awesome utility, it lets you count characters, words or lines very easily. wc -l counts lines in an input, since we're operating with one value per line we can easily count our hits and misses already:

$ wc -l hits-sorted.txt misses-sorted.txt
  132523 hits-sorted.txt
  220779 misses-sorted.txt
  353302 total

220779 / 132523 is a 1:1.66 ratio of hits to misses. That's not great…

Alright, now I'm also interested in how many unique URLs are hit versus missed. uniq tool deduplicates immediate sequences, so the input has to be sorted in order to deduplicate our entire file. We already did that. We can now count our urls with uniq < hits-sorted.txt | wc -l; uniq < misses-sorted.txt | wc -l. We get 49778 and 201178, respectively. It's to be expected that most of our cache misses would be in "rarer" urls; this gives us a 1:4 ratio of cached to uncached URL.

Let's say we want to dig down further into which URLs are most often hitting the cache, specifically. We can add -c to uniq in order to get a duplicate count in front of our URLs. To get the top ones at the top, we can then use sort, in reverse sort mode (-r), and it also needs to be numeric sort, not alphabetic (-n). head lets us get the top 10.

$ uniq -c < hits-sorted.txt | sort -nr | head
    815 /static/app/webfonts/fa-solid-900.woff2?d720146f1999
    793 /static/app/images/1.png
    786 /static/app/fonts/nunito-v9-latin-ext_latin-regular.woff2?d720146f1999
    760 /static/CACHE/js/output.cee5c4089626.js
    758 /static/images/crest/3/light/notfound.png
    757 /static/CACHE/css/output.4f2b59394c83.css
    756 /static/app/webfonts/fa-regular-400.woff2?d720146f1999
    754 /static/app/css/images/loading.gif?d720146f1999
    750 /static/app/css/images/prev.png?d720146f1999
    745 /static/app/css/images/next.png?d720146f1999

And same for misses:

$ uniq -c < misses-sorted.txt | sort -nr | head
     56 /
     14 /player/237678/
     13 /players/
     12 /teams/
     11 /players/top/
<snip>

So far this tells us static files are most often hit, and for misses it also tells us… something, but we can't quite track it down yet (and we won't, not in this post). We're not adjusting for how often the page is hit as a whole, this is still just high-level analysis.

One last thing I want to show you! Let's take everything we learned and analyze those URLs by prefix instead. We can cut our URLs again by slash with cut -d"/". If we want the first prefix, we can do -f1-2, or -f1-3 for the first two prefixes. Let's look!

cut -d'/' -f1-2 < hits-sorted.txt | uniq -c | sort -nr | head
 100189 /static
   5948 /es
   3069 /player
   2480 /fr
   2476 /es-mx
   2295 /pt-br
   2094 /tr
   1939 /it
   1692 /ru
   1626 /de

cut -d'/' -f1-2 < misses-sorted.txt | uniq -c | sort -nr | head
  66132 /static
  18578 /es
  17448 /player
  17064 /tr
  11379 /fr
   9624 /pt-br
   8730 /es-mx
   7993 /ru
   7689 /zh-hant
   7441 /it

This gives us hit-miss ratios by prefix. Neat, huh?

What have you been listening to this week? You don't need to do a 6000 word review if you don't want to, but please write something! If you've just picked up some music, please update on that as well, we'd love to see your hauls :)

Feel free to give recs or discuss anything about each others' listening habits.

You can make a chart if you use last.fm:

http://www.tapmusic.net/lastfm/

Remember that linking directly to your image will update with your future listening, make sure to reupload to somewhere like imgur if you'd like it to remain what you have at the time of posting.

This thread is posted weekly - please try to post all relevant US political content in here, such as news, updates, opinion articles, etc. Extremely significant events may warrant a separate topic, but almost all should be posted in here.

This is an inherently political thread; please try to avoid antagonistic arguments and bickering matches. Comment threads that devolve into unproductive arguments may be removed so that the overall topic is able to continue.

For the past month, I have been reading "The Wisdom of Psychopaths" by Kevin Dutton which delves into traits, behaviors, and motivations behind psychopaths. This book isn't just about serial killers but rather also the "successful" functional psychopaths such as stockbrokers, politicians, and business executives. You can read an excerpt from the book here if interested. A few interesting takeaways that I have had from the book so far are the innate cues that some people have on picking up on psychopathic cues. This is like speaking to someone and getting the heebie-jeebies from them for some reason. Apparently, women are more perceptive to this than men.

So, I'm curious if you have ever met a person that gave off that vibe, and what in particular gave you that vibe?

Sometimes Tildes users give people healthcare advice. Sometimes that advice disagrees with the advice already given by a qualified registered healthcare professional. That might be okay if the tildes advice was compliant with national guidance, but sometimes it isn't. Sometimes it's bad, dangerous, advice.

Should Tildes have rules about this?

Intermittently, for the past 15 years or so, logic has been an interest of mine. Back then I had trouble understanding exactly why certain things people said sounded so right/wrong, and how could I come up with proper responses.

Among others, in this time I've read one great book on informal logic (which I lost, unfortunately), quite a few articles, and studied the first chapters of the stupendous Gary Hardegree's symbolic logic.

Even though I love the subject, it is hard to sustain motivation alone. I wish to acquire a firmer grasp of logic and its applications to philosophy. Hence the suggestion of forming a study group.

It is my understanding that most Tilderinos are in STEM, especially areas surrounding computer science. So I anticipate that many users have an understanding of logic that greatly surpasses my own. Because of that, for some, a philosophical logic study group may seem too elementary to be of any value. Others may find it interesting to approach logic from a philosophical point of view.

In any case, the idea is to start from scratch. Besides the ability to read and write in the English language, no previous knowledge is required. No mathematics either.

I have two initial proposals.

1. An Illustrated Book of Bad Arguments

This one is ideal for a light, relaxed approach.

This awesome book describes 19 common logical fallacies using accessible language, with clear examples and suggestive illustrations. Not very technical, and a lot of it is well-known territory if you have an interest in logic. One chapter for each fallacy, each chapter is one page long. A great conversation starter.

2. Symbolic Logic: A First Course, by Gary Hardegree

I would choose this one myself. Hardegree is a wonderful teacher.

This book is one of the best teaching materials I have ever known, and surprisingly superior even to paid alternatives. A more proper introduction to logic. Hardegree is an excellent teacher, introducing concepts with precision in accessible language. The progression is smooth, you never feel that the exercises are either too easy or too hard. And there are plenty of exercises (with answers!) which are great for self-study.

We could start with either one of these books and follow from there. Just meeting once a week (or maybe biweekly) to discuss the chapter or chapter section we studied in that period.

I understand a lot of people like to do that kind of stuff on Discord, so that's a possibility.

From Wikipedia:

A cognitive bias is a systematic pattern of deviation from norm or rationality in judgment. Individuals create their own "subjective reality" from their perception of the input. An individual's construction of reality, not the objective input, may dictate their behavior in the world. Thus, cognitive biases may sometimes lead to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality.

For obvious reasons, it is much easier to identify biases in others than ourselves. Nevertheless, some of us went through practices (such as psychotherapy), experiences, and introspection that allowed us to put our biases in check. So, instead of scrutinizing the behavior of others (something that comes naturally to us, especially on the internet), here I ask you to exercise some self-criticism. What intellectual tendencies you have that obsessively repeat themselves in different contexts?

I should note that cognitive biases do not always lead to bad outcomes or falsehoods, as stated in Wikipedia:

Although it may seem like such misperceptions would be aberrations, biases can help humans find commonalities and shortcuts to assist in the navigation of common situations in life.[5]

On this thread, I am deliberately not asking about political bias or anything of the sort, including all the juicy controversial subjects surrounding it. Anything that often leads to uncivil discussion should be considered out of bounds.

For inspiration, look at this list (you don't need to identify a named bias, though... a subjective description of something you believe to be a form of bias is enough).

Dear Mods, due to the contentious nature of the subject, please feel free to act more aggressively on this topic than you currently do.

This topic is part of a series. It is meant to be a place for users to discuss creative projects they have been working on.

Projects can be personal, professional, physical, digital, or even just ideas.

If you have any creative projects that you have been working on or want to eventually work on, this is a place for discussing those.