16 votes

Tildes Issue Log - January 2020

8 comments

  1. [4]
    Deimos
    (edited )
    Link
    Thanks as always, Bauke. I wasn't sure if you'd include it in the Highlights, since from a user perspective it had no effect at all (and that was the goal), but this month I also did quite a bit...

    Thanks as always, Bauke.

    I wasn't sure if you'd include it in the Highlights, since from a user perspective it had no effect at all (and that was the goal), but this month I also did quite a bit of work to completely replace RabbitMQ in the site's stack with Redis streams for handling all the asynchronous/background tasks. This was the main commit, but there have been multiple other ones since then taking care of some additional things, and adding some more capabilities.

    There are some functional benefits to this switch (like making it so that tasks can still be queued even while a particular consumer isn't running), but overall the main nice thing was being able to eliminate an entire system and all of its dependencies. RabbitMQ required quite a bit in addition to itself (including Erlang), and it's really nice to have one less thing to install, update, monitor, etc. That makes me happy from the "simplicity" perspective, and it's great to have everything built on only PostgreSQL and Redis now.

    4 votes
    1. [2]
      VoidOutput
      Link Parent
      Small question to learn something new: I heard Postgres has a pub sub system included, is it not what you're using redis for? If it is, is Postgres' worse or are there other reasons to use redis?...

      Small question to learn something new: I heard Postgres has a pub sub system included, is it not what you're using redis for? If it is, is Postgres' worse or are there other reasons to use redis? If it is not, what is redis used for in tildes?

      4 votes
      1. Deimos
        (edited )
        Link Parent
        Yes, PostgreSQL has the LISTEN and NOTIFY commands for pubsub-type work. They're actually a core part of this and the source of everything that ends up in the streams: certain events in postgres...

        Yes, PostgreSQL has the LISTEN and NOTIFY commands for pubsub-type work. They're actually a core part of this and the source of everything that ends up in the streams: certain events in postgres trigger a NOTIFY, and I have a separate "PostgreSQL/Redis bridge" service that's running at all times that listens for all of those and translates them into events to store in the streams.

        You could use them stand-alone, but they're very limited and it would be difficult to duplicate a lot of the capabilities. All they do is send out a notification, and if nothing's listening at the time (which you can't tell), it's just lost. So if you had a consumer that crashes, any notifications sent while it's down will be missed. There are also various other capabilities that are handled nicely in Redis that would be somewhat tricky to reimplement in postgres, such as having multiple consumers for the same task.

        It's definitely possible to write a similar system in PostgreSQL alone (and I spent a while trying to think through how to do it), but since I'm already using Redis for other reasons and plan to keep it around, it was going to be a lot easier to just take advantage of streams instead of needing to reimplement everything they've already figured out (and probably doing a worse job of it). Redis is also used for:

        • Session storage
        • Rate-limiting actions, with a module that implements Generic Cell Rate Algorithm
        • Preventing users from using passwords seen in data breaches with a bloom filter
        5 votes
    2. Bauke
      Link Parent
      I decided not to as the RabbitMQ and Redis stuff largely goes over my head and I'd probably struggle in explaining it in a relatively simple way. :P If you want to write a bit about it I'll more...

      I decided not to as the RabbitMQ and Redis stuff largely goes over my head and I'd probably struggle in explaining it in a relatively simple way. :P If you want to write a bit about it I'll more than happily include it into the post. It's cool to read about the progress made even if it's just refactoring stuff.

      4 votes
  2. [2]
    Kuromantis
    Link
    Wow. Seriously though, you make this detailed log every month? That's impressive.

    The average time to close issues was 222.70 days or 5344.90 hours.

    Wow.

    Seriously though, you make this detailed log every month? That's impressive.

    4 votes
    1. Bauke
      Link Parent
      Thanks. The process is very largely automated, the only manual things are writing the Table of Contents, the Highlights section and filling out the official topics table. I've been meaning to...

      Thanks. The process is very largely automated, the only manual things are writing the Table of Contents, the Highlights section and filling out the official topics table. I've been meaning to restructure and redo some parts of the Issue Log for a while now so a lot more is automated and all I have to do is write the highlights, but I've been procrastinating doing other stuff. Maybe this month will be the one where I get around to it. (^:

      4 votes
  3. [2]
    suspended
    Link
    Have you posted this to r/tildes?

    Have you posted this to r/tildes?

    2 votes
    1. cfabbro
      (edited )
      Link Parent
      Nope... but I just did. Thanks again for still keeping this log up to date, @Bauke. And your new custom-font-size MR looks like it will be super cool! ~exemplary @Fuzzy Little Man Peach :P BTW for...

      Nope... but I just did.

      Thanks again for still keeping this log up to date, @Bauke. And your new custom-font-size MR looks like it will be super cool! ~‍exemplary @‍Fuzzy Little Man Peach :P

      BTW for anyone curious as to the delay, I was away from Tildes for the last week and so have a bit of catching up to do in ~tildes and elsewhere. Sorry to everyone for any delays in adding their suggestions to gitlab and not responding to @‍mentions/replies in that time... I will get to everything in the next few days.

      5 votes