Last month we started a community-maintained fork of the Tildes codebase. A lot has happened since then.

The biggest change: @Bauke and I have been added as maintainers to the official Tildes repo! As a result, we're moving the community fork to the backburner for now, as we focus on nearer-term changes that will directly improve the main website. Later on it's possible we'll pick up the fork again, where it will likely serve the purpose of self-hosting your own Tildes spinoff sites.

Deimos still has the final say on what makes it to the website. Bauke and I can't deploy changes directly. However, this arrangement is still much more streamlined than before, because we now have a lot more code review bandwidth for accepting outside contributions. Deimos has less work to do now: mostly testing out the live code on a staging server, and scanning over the code for security/privacy issues—but not full code reviews which often involve a lot of back-and-forth communication and reading and testing code.

What work have we done this past month?

It's mostly been setting up foundational stuff like configuring the GitLab repository, fixing the development environment, and writing docs.

More recently we have started fixing actual website bugs too: a bug when escaping a user mention (making sure \@talklittle doesn't turn into a link), and hiding <details> content in collapsed comments. Starting small but we've found a good rhythm and will work on more and bigger issues soon.

Big props to @Bauke for setting up a staging server! Currently at https://testing.tildes.community/ — This server will be instrumental in getting new code in a testable state in a live environment, which makes it easier to approve new features before deploying on the real Tildes site.

So we shouldn't submit code to the community fork?

No, please don't. We'll use the official Tildes repo from now on. I'll update last month's post to reflect this.

Is Docker support coming to the official repo?

Yes, very likely. Deimos has warmed up to the idea. Bauke and I have been using the Docker development environment and ironed out a lot of bugs this past month.

The official repo looks the same as before?

Our next steps are to port the community fork changes back upstream to the official repo. In addition to the master branch, we plan to add staging and develop branches. develop will be where development happens, while master will reflect what is currently deployed on Tildes.net.

How do I contribute to Tildes development?

Check this document: https://gitlab.com/tildes/tildes/-/blob/master/CONTRIBUTING.md

*Update (Feb 3, 2025): We've been added as maintainers on the official Tildes repo!

Much of the below is outdated now. Bauke and I will be helping out on the official Tildes repo instead, and the community fork is paused now.

See the new topic.

Original post below

It's happening: We're launching a community-maintained Tildes source code fork!

Link: https://gitlab.com/tildes-community/tildes-cf

@Bauke, as one of the top Tildes open source contributors, is on board as a co-maintainer, alongside myself. I hear @cfabbro is willing to help manage the issue tracker as well, continuing their long term efforts from the official repo.

Tildes' admin, @Deimos, has direct access to the repository as well. Although he is not expected to take an active role in maintaining this community fork, he will have visibility into everything going on with the fork.

Why?

Deimos has a lot going on outside of Tildes. We want to keep the Tildes codebase well maintained and remove some burden from him.

Back when he founded Tildes, Deimos was working as a fulltime unpaid volunteer on it, continuing that way for a few years. Not just code, but on everything administrative and financial; public relations, as in communicating officially inside the community and beyond; moderating the community; system administering the systems. Basically a ridiculous amount of effort for one person.

Now Tildes is a side project, and he has a day job, and there is not physically enough time for a (human, non-drug-reliant) owner to do all those things.

How will this new fork affect the Tildes website?

The hope is that Tildes can merge relevant changes back into the official upstream repository. If we implement things useful and desirable for Tildes, it should be possible to get those improvements onto the website.

Why not just add maintainers to the official repository?

There are some features that may be desirable for the community, but not relevant to Tildes itself. This includes things like a Docker development environment, which code contributors may find convenient, but are an extra maintenance burden on the official Tildes repo, as Tildes does not use Docker in any way (AFAIK).

Adding us to the official repository would also create a different dynamic, where there'd be an implicit endorsement by Deimos of all changes. This means the burden would essentially remain on the Tildes administrator to review, critique, and greenlight every single change. However, the entire point of this endeavor is that there isn't free bandwidth for that.

Also this fork opens up possibilities like making the code reusable for self-hosting entirely new websites based on the Tildes source code. While I don't personally have any specific plans regarding such, self-hosting has been a repeated request ever since Deimos open sourced Tildes years ago.

Is "Tildes Community Fork" good enough of a name?

Thanks for reading this far! The fork needs a name. It will live in the "Tildes Community" GitLab group at https://gitlab.com/tildes-community/.

For now I've simply called it "Tildes Community Fork" and put it at https://gitlab.com/tildes-community/tildes-cf.

Any better naming ideas? It's not too late to change.

Next steps: We'll start migrating GitLab issues over

I think we're ready to start copying any "low-hanging fruit" issues from the official issues to the new community fork issues. If you have an issue you think qualifies as such, especially if it was ever labeled as "Approved" in the past, please feel free to copy it to the new issue tracker. Please link back to the original too.

It's still a side project for us

Please keep in mind it's still a side project for us. Although we're excited to push the project forward, please keep expectations in check. We're doing this as volunteers. Please be polite and don't rush us!

Preface: I am familiar with Habitica. This idea would probably scratch a similar itch, but I'm also using this as an opportunity to sharpen my Rust skills.

My idea came about when I was trying to find out some new tactics to curb my alcohol consumption, which isn't quite out of control yet, but I don't want to tempt fate.

I've also really liked the progression aspect of RPGs. What if I could gamify my quest to not drink alcohol and make it sort of a fun, unique RPG experience at the same time?

In the broadest sense, it would go something like this:

• You open the game up, ideally each day. You are instantly prompted: "Did you drink yesterday?" (and perhaps it will go back a few more days if you skipped).
• For each day you answer "no", you are rewarded with some sort of tokens, credits, etc. -- currency to play the game. If you answer "yes", maybe you get penalized somehow.
• Then, you pick up your journey, which is sort of a standard RPG experience -- fighting battles, buying gear, learning spells, leveling up, advancing through the world, you name it.
• The game should get progressively more difficult, but should not have an ending, as "quitting alcohol" does not have an ending either. At the same time, it should scratch the RPG progression itch.

The initial game concept I came up with is just one that I see as the quickest way to get this off the ground, which would be something CLI-based, where you are presented with a menu ("visit shop, enter arena, view equipment" etc.). You spend battle tokens to enter into arena battles, which reward experience points, money, and gear. You level up, work towards a build (there needs to be a way to respec because restarting isn't really an option), and progress through the arena.

In total, you would probably spend less than 5 minutes every day playing the game, which is by design. It should be an every day habit. But, there should be enough entertainment value that, if I'm not getting those sweet battle tokens by not drinking, I'm missing out on experiencing the game (or, I could lie, which defeats the purpose of the app).

So that's where I'm at right now. I'm really interested to hear your thoughts, ideas, critiques, etc. before I spend a free weekend building out a concept.

Some questions in particular:

• I was leaning toward just building this in CLI because it will be extremely simple. It could just be a matter of STDINs. However, I'm open to other Rust-based options. Is there a good Rust UI toolkit or web framework that is worth looking into that would make this a little more modern?
• What about game features? What could make this a really fun experience, while also balancing the whole concept of being built around your life and your habits?

In the end, this is a deeply personal project that would be built, first and foremost, for my specific needs. But that's not to say I couldn't build it with some scalability in mind. Rather than asking about alcohol, perhaps the "habits" can be customized, and so forth.

Anyway, have a great weekend!

Hello everyone.

I am planning to get back into Linux development after working with Mac only for almost a decade. On Mac, one of the most important lessons that I learned was to always use Homebrew. Using various package managers (e.g. Homebrew, NPM, Yarn, Pip, etc.) creates situations in which you don't know how to uninstall or upgrade certain pieces of software. Also, it's hard to generate a complete overview.

How do you Linux folks handle this?

Bonus question: How do you manage your dotfiles securely? I use Bitwarden, and it's a bit clunky.

If that helps, I want to try Mint and always use Oh My ZSH!.

Hi,

I'm curious if anyone in this group has achieved success in game development, whether that's carving out a career or earning any amount of income from it.

I'm currently working as a software developer, but my passion lies in game development. I'm all too aware that achieving any measure of success in this field is next to impossible. Hence, I'm reaching out here, hoping to gather insights and advice from those who have walked this path in the past, or those who are currently walking alongside/behind me.

One of my specific questions is about the types of games I should focus on creating. Specifically, I've heard differing opinions on whether it's more advantageous to develop a series of small games with advertisements for mobile platforms or to invest in larger, premium games for platforms like Steam. Can anyone share their insights or experiences regarding this dilemma? Is there a clear advantage to one approach over the other?

Currently I am using godot to make a larger scale game, but I am considering switching to defold and making smaller scale games with ads.

I saw some folks here discuss making games for the playdate. How much should one consider targeting niche platforms like this? Some of the users I saw discuss this seem to have had good success.

Some general questions: How did you break into game dev? What were you doing before? Do you see game dev as a viable career, only as a source of side income, or is it just a hobby?

Any guidance or experiences you can share would be greatly appreciated.

Web devs: what's up with this trend? For enterprise apps, I get it…single sign-on needs to detect what your email domain is to send you to your identity provider. For consumers, I feel like it's gotta be one of these reasons:

• Users don't know about the tab key being able to move to other fields on a page
• Mobile users don't really have a tab key, despite there being "previous/next field" arrows on the stock iOS keyboard since its inception (Android users, help me out please)
• Users tend to hit Enter after typing in their username, leading to a form submission with a blank password
• Security, maybe? In the past I have sent a link and a password in separate emails or separate communication methods entirely. Are you hashing/salting these separately for better MITM mitigation?

Did your UX team make a decision? Are my password managers forever doomed to need a "keyboard combo" value for every entry from now on?

Non-devs: do you prefer one method over the other? If so, why?

Tildes maintainers: selfishly, thanks for keeping these together :)

I've been trying to set up a dev enviornment for Tildes, mainly so that I can actually test my MR (!136), and I've been running into a few issues.

However, since we also have a new influx of people who might be interested in contributing to Tildes, it seems like a good time to collect resources on setting up the dev environment, as well as helping anyone running into issues.

So, if you have issues or advice, post them here! I'll be adding my questions in a comment shortly.

Relevant wiki pages:

• Development / Initial Setup
• Development / General Development Info

Edit: A more recent post on setting up the dev environment on Apple Silicon / M1 Macs

I was fortunate enough to grab a discounted 4090 while on my travels and just got everything installed. Already having a lot of fun pumping all my games to max settings, but I'm also interested in running generative AI stuff locally to really take advantage of all that VRAM.

Do you have any newbie-friendly Windows 11 software to recommend for getting started? Thanks!

Hey Tildes, with the renewed interest in the site, it got me thinking that we should hold a fundraiser for the not-for-profit company—which currently consists of just one person—that runs Tildes. It's overdue.

Disclaimer: These are my words as a member of the community. I haven't run this message by the admin before posting. I may have gotten some details wrong.

Where to donate

• GitHub Sponsors: https://github.com/sponsors/Deimos
• Patreon: https://www.patreon.com/tildes

History

A bit of history: The site admin, @Deimos ran the first three years of the site working full-time on it, paid only by donations, plus a $5000 GitHub sponsor match one year, which I'm not even sure was fully achieved, or only just barely.

For that time period 2018-2020, a lowball salary as a software engineer with his experience would have been $100,000 USD per year not including benefits.

If he received $5000 in donations per year (almost certainly an overestimate for more recent years) plus the $5000 GitHub match for the first year—for the 5 years of Tildes' life, that's about $30,000.

The remaining opportunity cost of $270,000 was essentially paid out of pocket by himself, as a donation to the community. Plus remember there are server expenses, legal incorporation expenses, etc. And, y'know, rent.

In recent years he had to take a full-time job because the situation was, of course, unsustainable.

App?

I announced in April that a mobile app is under development. Originally, I was planning to take my time and release a first alpha by the end of 2023.

How about if we struck a deal: get the donation numbers up and I will devote more time to the app, as opposed to splitting my time between it and contract work and other projects.

What's the deal?

• 150 active donors combined on GitHub Sponsors and Patreon—I'll release an alpha by November.
  GOAL REACHED
• 300 active donors—I'll release an alpha by October.
  GOAL REACHED
• 500 active donors—I'll release an alpha by September.

The dollar amounts don't matter.

As of writing, we are at 46 active donors.

What's in it for you, though?

Feeling like I did a good deed, I guess? I'm not looking for a "slice of the pie," to be clear. In some sense I'd be matching your donations with my time, aka opportunity cost.

If I donate, can I bother the admin to work more on the site?

No.

Again, I haven't run this fundraiser by the admin. He will certainly keep his full-time employment for the foreseeable future, and will not magically have more hours in the day to devote to Tildes.

With a sustainable budget, though, a lot can happen in the future. Contracting out work to others, for example.

But the point of this fundraiser is more to make a small dent in the past debt we owe the admin, not making any promises whatsoever on the future of the site and how it's run.

Let's go, my fellow Tilderinos!

• GitHub Sponsors: https://github.com/sponsors/Deimos
• Patreon: https://www.patreon.com/tildes

Bevy just released their version 0.11, so I figured it would be a nice opportunity to ask the Tildes gamedevs if they were using it :)

Bevy is a rust game engine - more like a set of libraries actually - that's been gaining popularity the last few years. It has become the de facto toolset if you want to make a game in rust. It is very opinionated towards Entity-Component-System (ECS), and uses the pattern to facilitate parallelism and multi-threading.

Personally, I'm using the bevy-ecs lib (not the whole engine) to write a roguelike and hone my skills in rust. I enjoy it but it's not really beginner-friendly. The official docs are lacking, and you'll have to dig in the auto-generated api docs to make the most out of it. However, I appreciate that each release not only brings new features, but also refines existing ones. The engine is getting better - not only bigger - release after release.

Want to contribute a theme to Tildes but don't know where to start? Let's fix that.

Before we start, get yourself a development environment setup and do a quick read through of the general development info to get acquainted with how Tildes works (or at least the HTML and CSS section).

For this walkthrough I'll be using tildexample as the example name for the theme, but if you decide to contribute a theme for real, make sure it uses the proper name of your theme. :P

Step 1: Sassy _Sass

Open the Tildes codebase using your text editor of choice and navigate to the themes directory at tildes/scss/themes. Then create a copy of _default.scss at _tildexample.scss. The default White theme is the canonical source of all colors used, so it's the best place to start from.

Below is an annotated example of all the things you need to change in your new theme file.

// Add a small description of the theme here with maybe a link to its website.
// Check the other themes for examples. https://example.org/tildexample

// Change the theme variable to $theme-tildexample
// ↓ ↓ ↓ ↓ ↓ ↓ 
$default-theme: (
  // A whole bunch of color definitions, edit as your theme demands.
  // ...
);

// Append ".theme-tildexample" to the body selector.
// ↓ ↙
body {
  // Don't forget to update the theme variable here too.
  //                  ↓ ↓ ↓ ↓ ↓ ↓ ↓
  @include use-theme($default-theme);
}

@include theme-preview-block(
  // Change the text to tildexample.
  // ↓ ↓
  "white",
  // And again update the theme variable here.
  //       ↓ ↓ ↓ ↓ ↓ ↓ ↓
  map-get($default-theme, "foreground-primary"),
  map-get($default-theme, "background-primary")
  //       ↑ ↑ ↑ ↑ ↑ ↑ ↑
);

Once that's done, head to tildes/scss/styles.scss and at the bottom of the file add your theme import:

@import "themes/tildexample";

Step 2: Hardcoding a TheMe coLor

Boy that title is a stretch just to say, we need to add 2 lines to the HTML base template.

Inside the tildes/tildes/templates/base.jinja2 file is a section of if/elif/elif/elif/... statements to set the theme color meta element. Add yourself an elif block and add your theme color.

For this you probably want to use the background-primary color you used in your theme definition. I've used #ff00dd below because it spells food. I'm such a jokester.

{% elif request.current_theme == "tildexample" %}
<meta name="theme-color" content="#ff00dd">
{% endif %}

Step 3: Snakey Wakey

Finally the last step is to grab your trusty pungi and give it a blow.

Head to tildes/tildes/views/settings.py and find the THEME_OPTIONS constant. Here you want to add the theme class you used in body.theme-<this part> and a proper name that will be shown in the theme dropdown.

THEME_OPTIONS = {
    "white": "White",
    # Many other themes...
    "tildexample": "Tildes Theme Example",
}

Once that's all been done, check it out in your development site and see if it works.

Now git!

Commit. Push. Merge request. Have some water. Deimos reviews, merges and deploys your theme. Job's done.

If I have a quick Tildes dev question, where should I ask it?

Been lurking for a week on tildes now and I am really glad this place exists. The crow here is exactly what I have been missing on Reddit for a while now.

Having said that, the whole Reddit situation has some-what motivated me to get the balls rolling on an idea that I have had for a while and I am looking for advice on the same.

I have often heard this phrase "Learn programming by building" but whenever I dive in to the resources, I fall flat due to the information overload and the general abstractness that the field has (I appreciate abstractness but here it demotivates me) and I have never found a proper resource that I could follow to actually build something instead of just blindly following tutorials and playing with them.

So, my question is how do I translate "learn by building a project" into a practical framework.

I know of 100 days of swift and I really like that approach however I don't think I want to start with swift or build an iOS app right now.

I'm having some trouble getting the local developer environment set up on Apple hardware, specifically it seems because Virtualbox, which is used as the provider, is not properly functioning on Apple hardware.

Is there anyone here who has managed to get it up and running?

Someone mentioned elsewhere that they signed up for Tildes “years ago during the beta”. That reminded me: Tildes hasn’t reached beta-testing yet.

Officially, Tildes is still in alpha-testing phase.

The login page says “Tildes is currently in invite-only alpha...” And the Contact page says “To request an invite to the Tildes alpha...”

We’re still in alpha-testing. Alpha-testing of software usually happens on an incomplete product before it is released to the customer.

This is a very important point. Tildes is not feature-complete yet: there are literally hundreds of feature requests yet to work on before Tildes will be what people want it to be – and even that list is far from complete. In Agile software development terms, Tildes is a minimum viable product, or, in other words, “a version of a product with just enough features to be usable by early customers who can then provide feedback for future product development”.

Tildes works as it is, but it’s a bare-bones forum: you can post, and comment… and that’s about it. It’s a proof of concept. There are a few minor tweaks here and there, which give the impression that Tildes is more complex than it is, but they’re misleading. There are plans to make Tildes a more complex website but, right now, most of that complexity of Tildes exists only in people’s imaginations (and there have been some very imaginative people contributing to that list of future features!).

Most questions about “Why doesn’t Tildes do X?” or “Can Tildes do X?” can be answered simply by saying “Tildes is incomplete and X hasn’t been built yet.” There are some questions about missing features which can be answered by saying “Tildes was never intended to do X”, but those are far and away in the minority. Most flaws, drawbacks, and problems with Tildes exist because Tildes is still a proof of concept, rather than a finished product.

It’s also worth noting that Tildes’ current feature set is absolutely not up to the task if the user base and site activity increase too quickly. There’s too much manual tinkering required at the moment to make things work properly: for one thing, there are no significant moderation tools on Tildes (that’s almost all done manually at the moment). There are still a lot of features yet to be built - and we don’t even know what some of those features are yet!

To pre-empt the people who will rightly point out that Tildes is 5 years old: Tildes’ feature set was intended to grow gradually over time, in line with a gradual growth in users, activity, and the need for those features to exist. However, Tildes has not undergone much growth over the past few years, so the existing features were sufficient to manage the existing activity. Basically, the site didn’t need a lot of fancy features to handle the low traffic here.

This sudden surge of new users might change that. But it will take time to build more features. That was always the intention, and it hasn’t changed now.

Until then: Tildes is still in alpha-testing. It’s an unfinished product. Set your expectations accordingly.

EDIT:

If you're one of the many people who seem to be replying to this topic, saying "it's okay, I like this simple bare-bones site as it is"... then you're probably not part of my original intended audience for this topic. This topic was aimed at all the people who are arriving here, being taken aback at how simple Tildes is, and wondering where the advanced features are.

However, we can still take the "set your expectations accordingly" message and apply it to you: "Tildes is an unfinished product, so you can expect it to change in the future. It won't be like this forever."

Either way, "set your expectations accordingly" is the message here, whether you're expecting more features and not finding them, or whether you're expecting simplicity and enjoying what you see. Either way, you should know that things will change around here. Maybe slowly. Maybe quickly. Maybe they'll get better from your point of view. Maybe they'll get worse from your point of view. But, change they will.

I'm thinking about increasing the level of my open source contributions a bit. Instead of searching blindly until I stumble upon an issue that:
A) Piques my interest
B) I feel somewhat qualified to implement

I figured I'd check with the tildes community. Is there any Open Source software that you use that is missing a feature/capability? Can you give a brief description of it (bonus points for links to an issue tracker with an open ticket :))?

Can't of course promise anything will come of it, but if I do pick up your suggestion at least I'll give you a ping if I make any progress!

They ask for years of experience, skills that no Jr would know since, well, it is a Jr and the process to apply for jobs are surreal. Thousands of tests, interviews that goes nowhere and lots of ghosting. And the pay is not that good. No wonder after 2 or 3 years of experience a lot of develpers starts working for companies outside of Brazil.

Last one to contact me sent me a test to do it in 1 week. I went above and beyond and learned a lot of things. Before this, i had some small projects in Go and Python. Now i needed to learn Docker, tests, github actions, Postgresql and other things. Not everything was mandatory, but i did my best and did it all. I finished in 5 days since i have a day job.

Here is the result: https://github.com/crdpa/conservice

Showing the data in the browser was not necessary, but i think it was a nice touch and well made. If this does not land me a job as a junior developer i don't know what else could.

I'm glad i already have a job in another area, but me and my SO are separated by a 4 hour drive and i'm tired. I want to work from home to be near her and our dog. Paying rent in two places is becoming a burden.

I would be happy if you guys could test the application i made. It only needs docker.

And do you guys have any tips from now on?

This below is a summary of some real-world performance investigation I recently went through. The tools I used are installed on all linux systems, but I know some people don't know them and would straight up jump to heavyweight log analysis services and what not, or writing their own solution.

Let's say you have request log sampling in a bunch of log files that contain lines like these:

127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000
127.0.0.1 [2021-05-27 23:51:22.019] "GET /pl/player/123456/changelog/ HTTP/1.1" 200 16524 TLSv1.2 MISS-CLUSTER SessionID:(null) Cache:

You might recognize Fastly logs there (IP anonymized). Now, there's a lot you might care about in this log file, but in my case, I wanted to get a breakdown of hits vs misses by URL.

So, first step, let's concatenate all the log files with cat *.log > all.txt, so we can work off a single file.

Then, let's split the file in two: hits and misses. There are a few different values for them, the majority are covered by either HIT-CLUSTER or MISS-CLUSTER. We can do this by just grepping for them like so:

grep HIT-CLUSTER all.txt > hits.txt; grep MISS-CLUSTER all.txt > misses.txt

However, we only care about url and whether it's a hit or a miss. So let's clean up those hits and misses with cut. The way cut works, it takes a delimiter (-d) and cuts the input based on that; you then give it a range of "fields" (-f) that you want.

In our case, if we cut based on spaces, we end up with for example: 127.0.0.1 [2021-05-27 23:28:34.460] "GET /static/images/flags/2/54@3x.webp HTTP/2" 200 1806 TLSv1.3 HIT-CLUSTER SessionID:(null) Cache:max-age=31536000.

We care about the 5th value only. So let's do: cut -d" " -f5 to get that. We will also sort the result, because future operations will require us to work on a sorted list of values.

cut -d" " -f5 hits.txt | sort > hits-sorted.txt; cut -d" " -f5 misses.txt | sort > misses-sorted.txt

Now we can start doing some neat stuff. wc (wordcount) is an awesome utility, it lets you count characters, words or lines very easily. wc -l counts lines in an input, since we're operating with one value per line we can easily count our hits and misses already:

$ wc -l hits-sorted.txt misses-sorted.txt
  132523 hits-sorted.txt
  220779 misses-sorted.txt
  353302 total

220779 / 132523 is a 1:1.66 ratio of hits to misses. That's not great…

Alright, now I'm also interested in how many unique URLs are hit versus missed. uniq tool deduplicates immediate sequences, so the input has to be sorted in order to deduplicate our entire file. We already did that. We can now count our urls with uniq < hits-sorted.txt | wc -l; uniq < misses-sorted.txt | wc -l. We get 49778 and 201178, respectively. It's to be expected that most of our cache misses would be in "rarer" urls; this gives us a 1:4 ratio of cached to uncached URL.

Let's say we want to dig down further into which URLs are most often hitting the cache, specifically. We can add -c to uniq in order to get a duplicate count in front of our URLs. To get the top ones at the top, we can then use sort, in reverse sort mode (-r), and it also needs to be numeric sort, not alphabetic (-n). head lets us get the top 10.

$ uniq -c < hits-sorted.txt | sort -nr | head
    815 /static/app/webfonts/fa-solid-900.woff2?d720146f1999
    793 /static/app/images/1.png
    786 /static/app/fonts/nunito-v9-latin-ext_latin-regular.woff2?d720146f1999
    760 /static/CACHE/js/output.cee5c4089626.js
    758 /static/images/crest/3/light/notfound.png
    757 /static/CACHE/css/output.4f2b59394c83.css
    756 /static/app/webfonts/fa-regular-400.woff2?d720146f1999
    754 /static/app/css/images/loading.gif?d720146f1999
    750 /static/app/css/images/prev.png?d720146f1999
    745 /static/app/css/images/next.png?d720146f1999

And same for misses:

$ uniq -c < misses-sorted.txt | sort -nr | head
     56 /
     14 /player/237678/
     13 /players/
     12 /teams/
     11 /players/top/
<snip>

So far this tells us static files are most often hit, and for misses it also tells us… something, but we can't quite track it down yet (and we won't, not in this post). We're not adjusting for how often the page is hit as a whole, this is still just high-level analysis.

One last thing I want to show you! Let's take everything we learned and analyze those URLs by prefix instead. We can cut our URLs again by slash with cut -d"/". If we want the first prefix, we can do -f1-2, or -f1-3 for the first two prefixes. Let's look!

cut -d'/' -f1-2 < hits-sorted.txt | uniq -c | sort -nr | head
 100189 /static
   5948 /es
   3069 /player
   2480 /fr
   2476 /es-mx
   2295 /pt-br
   2094 /tr
   1939 /it
   1692 /ru
   1626 /de

cut -d'/' -f1-2 < misses-sorted.txt | uniq -c | sort -nr | head
  66132 /static
  18578 /es
  17448 /player
  17064 /tr
  11379 /fr
   9624 /pt-br
   8730 /es-mx
   7993 /ru
   7689 /zh-hant
   7441 /it

This gives us hit-miss ratios by prefix. Neat, huh?

I'm attempting to provision two mirror staging and production environments for a future SaaS application that we're close to launching as a company, and I'd like to get some feedback on the provisioning script I've created that takes a default VPS from our hosting provider, DigitalOcean, and readies it for being a secure hosting environment for our application instance (which runs inside Docker, and persists data to an unrelated managed database).

I'm sticking with a simple infrastructure architecture at the moment: A single VPS which runs both nginx and the application instance inside a containerised docker service as mentioned earlier. There's no load balancers or server duplication at this point. @Emerald_Knight very kindly provided me in the Tildes Discord with some overall guidance about what to aim for when configuring a server (limit damage as best as possible, limit access when an attack occurs)—so I've tried to be thoughtful and integrate that paradigm where possible (disabling root login, etc).

I’m not a DevOps or sysadmin-oriented person by trade—I stick to programming most of the time—but this role falls to me as the technical person in this business; so the last few days has been a lot of reading and readying. I’ll run through the provisioning flow step by step. Oh, and for reference, Ubuntu 20.04 LTS.

First step is self-explanatory.

#!/bin/sh

# Name of the user to create and grant privileges to.
USERNAME_OF_ACCOUNT=

sudo apt-get -qq update
sudo apt install -qq --yes nginx
sudo systemctl restart nginx

Next, create my sudo user, add them to the groups needed, require a password change on first login, then copy across any provided authorised keys from the root user which you can configure to be seeded to the VPS in the DigitalOcean management console.

useradd --create-home --shell "/bin/bash" --groups sudo,www-data "${USERNAME_OF_ACCOUNT}"
passwd --delete $USERNAME_OF_ACCOUNT
chage --lastday 0 $USERNAME_OF_ACCOUNT

HOME_DIR="$(eval echo ~${USERNAME_OF_ACCOUNT})"
mkdir --parents "${HOME_DIR}/.ssh"
cp /root/.ssh/authorized_keys "${HOME_DIR}/.ssh"

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown --recursive "${USERNAME_OF_ACCOUNT}":"${USERNAME_OF_ACCOUNT}" "${HOME_DIR}/.ssh"

sudo chmod 775 -R /var/www
sudo chown -R $USERNAME_OF_ACCOUNT /var/www
rm -rf /var/www/html

Installation of docker, and run it as a service, ensure the created user is added to the docker group.

sudo apt-get install -qq --yes \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88

sudo add-apt-repository --yes \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

sudo apt-get -qq update
sudo apt install -qq --yes docker-ce docker-ce-cli containerd.io

# Only add a group if it does not exist
sudo getent group docker || sudo groupadd docker
sudo usermod -aG docker $USERNAME_OF_ACCOUNT

# Enable docker
sudo systemctl enable docker

sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version

Disable root logins and any form of password-based authentication by altering sshd_config.

sed -i '/^PermitRootLogin/s/yes/no/' /etc/ssh/sshd_config
sed -i '/^PasswordAuthentication/s/yes/no/' /etc/ssh/sshd_config
sed -i '/^ChallengeResponseAuthentication/s/yes/no/' /etc/ssh/sshd_config

Configure the firewall and fail2ban.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw reload
sudo ufw --force enable && sudo ufw status verbose

sudo apt-get -qq install --yes fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Swapfiles.

sudo fallocate -l 1G /swapfile && ls -lh /swapfile
sudo chmod 0600 /swapfile && ls -lh /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile && sudo swapon --show
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Unattended updates, and restart the ssh daemon.

sudo apt install -qq unattended-upgrades
sudo systemctl restart ssh

Some questions

You can assume these questions are cost-benefit focused, i.e. is it worth my time to investigate this, versus something else that may have better gains given my limited time.

1. Obviously, any critiques of the above provisioning process are appreciated—both on the micro level of criticising particular lines, or zooming out and saying “well why don’t you do this instead…”. I can’t know what I don’t know.

2. Is it worth investigating tools such as ss or lynis (https://github.com/CISOfy/lynis) to perform server auditing? I don’t have to meet any compliance requirements at this point.

3. Do I get any meaningful increase in security by implementing 2FA on login here using google authenticator? As far as I can see, as long as I'm using best practices to actually ssh into our boxes, then the likeliest risk profile for unwanted access probably isn’t via the authentication mechanism I use personally to access my servers.

4. Am I missing anything here? Beyond the provisioning script itself, I adhere to best practices around storing and generating passwords and ssh keys.

Some notes and comments

1. Eventually I'll use the hosting provider's API to spin up and spin down VPS's on the fly via a custom management application, which gives me an opportunity to programmatically execute the provisioning script above and run some over pre- and post-provisioning things, like deployment of the application and so forth.

2. Usage alerts and monitoring is configured within DigitalOcean's console, and alerts are sent to our business' Slack for me to action as needed. Currently, I’m settling on the following alerts:
   1. Server CPU utilisation greater than 80% for 5 minutes.
   2. Server memory usage greater than 80% for 5 minutes.
   3. I’m also looking at setting up daily fail2ban status alerts if needed.