Any other developers also strongly resistant to adding secondary data stores to their software?

teaearlgraycold
2024-01-30T17:35:05Z

I'm currently building an MVP for a startup, solo. We've got Postgres pulling triple duty as the go-to database for all normal relational data, a vector database with pgvector, and a job queue (With the magic of SELECT ... FROM "Jobs" WHERE ... FOR UPDATE SKIP LOCKED LIMIT 1). Every time I go out looking for solutions to problems it feels like the world really wants me to get a dedicated vector store or to use Redis as a job queue.

Back when I was a Rails developer a good majority of the ActiveJob implementers used Redis. Now that I'm doing NodeJS the go-to is Bull which can only serialize jobs to Redis. They back this with claims that I can scale to thousands of jobs per second! I have to assume this theoretical throughput benefit from using Redis is utilized by 0.01% of apps running Bull.

So I ended up implementing a very simple system. Bull wouldn't have been a good fit anyway as we have both Python and Typescript async workers, so a simple system that I fully understand is more useful at the moment. I'm curious who else shares my philosophy.

Edit: I'll try to remember to update everyone in a year with the real world consequences of my design choices.

Comments URL: https://tildes.net/~comp/1dvh/any_other_developers_also_strongly_resistant_to_adding_secondary_data_stores_to_their_software

Votes: 16

Comments: 34