Thanks as always, Bauke. I wasn't sure if you'd include it in the Highlights, since from a user perspective it had no effect at all (and that was the goal), but this month I also did quite a bit...
Thanks as always, Bauke.
I wasn't sure if you'd include it in the Highlights, since from a user perspective it had no effect at all (and that was the goal), but this month I also did quite a bit of work to completely replace RabbitMQ in the site's stack with Redis streams for handling all the asynchronous/background tasks. This was the main commit, but there have been multiple other ones since then taking care of some additional things, and adding some more capabilities.
There are some functional benefits to this switch (like making it so that tasks can still be queued even while a particular consumer isn't running), but overall the main nice thing was being able to eliminate an entire system and all of its dependencies. RabbitMQ required quite a bit in addition to itself (including Erlang), and it's really nice to have one less thing to install, update, monitor, etc. That makes me happy from the "simplicity" perspective, and it's great to have everything built on only PostgreSQL and Redis now.
Small question to learn something new: I heard Postgres has a pub sub system included, is it not what you're using redis for? If it is, is Postgres' worse or are there other reasons to use redis?...
Small question to learn something new: I heard Postgres has a pub sub system included, is it not what you're using redis for? If it is, is Postgres' worse or are there other reasons to use redis? If it is not, what is redis used for in tildes?
Yes, PostgreSQL has the LISTEN and NOTIFY commands for pubsub-type work. They're actually a core part of this and the source of everything that ends up in the streams: certain events in postgres...
Yes, PostgreSQL has the LISTEN and NOTIFY commands for pubsub-type work. They're actually a core part of this and the source of everything that ends up in the streams: certain events in postgres trigger a NOTIFY, and I have a separate "PostgreSQL/Redis bridge" service that's running at all times that listens for all of those and translates them into events to store in the streams.
You could use them stand-alone, but they're very limited and it would be difficult to duplicate a lot of the capabilities. All they do is send out a notification, and if nothing's listening at the time (which you can't tell), it's just lost. So if you had a consumer that crashes, any notifications sent while it's down will be missed. There are also various other capabilities that are handled nicely in Redis that would be somewhat tricky to reimplement in postgres, such as having multiple consumers for the same task.
It's definitely possible to write a similar system in PostgreSQL alone (and I spent a while trying to think through how to do it), but since I'm already using Redis for other reasons and plan to keep it around, it was going to be a lot easier to just take advantage of streams instead of needing to reimplement everything they've already figured out (and probably doing a worse job of it). Redis is also used for:
Nope... but I just did. Thanks again for still keeping this log up to date, @Bauke. And your new custom-font-size MR looks like it will be super cool! ~exemplary @Fuzzy Little Man Peach :P BTW for...
Thanks again for still keeping this log up to date, @Bauke. And your new custom-font-size MR looks like it will be super cool! ~exemplary @Fuzzy Little Man Peach :P
BTW for anyone curious as to the delay, I was away from Tildes for the last week and so have a bit of catching up to do in ~tildes and elsewhere. Sorry to everyone for any delays in adding their suggestions to gitlab and not responding to @mentions/replies in that time... I will get to everything in the next few days.
Thanks as always, Bauke.
I wasn't sure if you'd include it in the Highlights, since from a user perspective it had no effect at all (and that was the goal), but this month I also did quite a bit of work to completely replace RabbitMQ in the site's stack with Redis streams for handling all the asynchronous/background tasks. This was the main commit, but there have been multiple other ones since then taking care of some additional things, and adding some more capabilities.
There are some functional benefits to this switch (like making it so that tasks can still be queued even while a particular consumer isn't running), but overall the main nice thing was being able to eliminate an entire system and all of its dependencies. RabbitMQ required quite a bit in addition to itself (including Erlang), and it's really nice to have one less thing to install, update, monitor, etc. That makes me happy from the "simplicity" perspective, and it's great to have everything built on only PostgreSQL and Redis now.
Small question to learn something new: I heard Postgres has a pub sub system included, is it not what you're using redis for? If it is, is Postgres' worse or are there other reasons to use redis? If it is not, what is redis used for in tildes?
Yes, PostgreSQL has the
LISTENandNOTIFYcommands for pubsub-type work. They're actually a core part of this and the source of everything that ends up in the streams: certain events in postgres trigger a NOTIFY, and I have a separate "PostgreSQL/Redis bridge" service that's running at all times that listens for all of those and translates them into events to store in the streams.You could use them stand-alone, but they're very limited and it would be difficult to duplicate a lot of the capabilities. All they do is send out a notification, and if nothing's listening at the time (which you can't tell), it's just lost. So if you had a consumer that crashes, any notifications sent while it's down will be missed. There are also various other capabilities that are handled nicely in Redis that would be somewhat tricky to reimplement in postgres, such as having multiple consumers for the same task.
It's definitely possible to write a similar system in PostgreSQL alone (and I spent a while trying to think through how to do it), but since I'm already using Redis for other reasons and plan to keep it around, it was going to be a lot easier to just take advantage of streams instead of needing to reimplement everything they've already figured out (and probably doing a worse job of it). Redis is also used for:
Wow.
Seriously though, you make this detailed log every month? That's impressive.
Nope... but I just did.
Thanks again for still keeping this log up to date, @Bauke. And your new custom-font-size MR looks like it will be super cool! ~exemplary @Fuzzy Little Man Peach :P
BTW for anyone curious as to the delay, I was away from Tildes for the last week and so have a bit of catching up to do in ~tildes and elsewhere. Sorry to everyone for any delays in adding their suggestions to gitlab and not responding to @mentions/replies in that time... I will get to everything in the next few days.