I’d like your help – and your stories!

I’m working on an article with a tentative title of “Tales of the Crypt: Horror stories (about your past) where database performance caused a real problem.” It’s meant to be schadenfreude nostalgia, about your late nights coping with a performance issue (with, hopefully, a happy ending of “…and this is what we did to fix it”).

So, what happened? Tell me about it.

I do want to quote you, but we can be oblique about the attribution – especially because sometimes these stories are from a previous employer and do not represent any current affiliation. But I do want the verisimilitude that demonstrates that these tales-of-woe come from real people. As a result, I’m fine with writing, “Kim’s first job was as a mainframe programmer at a hotel chain, where database transactions required tape changes. ‘Yada yada story,’ says Kim, who now is CIO of a Midwest insurance firm.” Real person, but you don’t need to worry about getting anyone to approve your words. (Though if you’re happy with full name, company, and role, I’m even happier; send in a private message if you prefer.)

I used an ancient example above, but I’m hoping for more recent database performance stories. Ideally some of the “here’s how we fixed it” become practical suggestions for developers who are enduring such a situation today.

I recently discovered how much of a hassle it is to migrate off of Heroku Postgres. Both to keep the user from screwing themselves and as a vendor lock-in mechanism there's no possibility of running an external replica for your Heroku Postgres database. Unless I'm missing something (after a bunch of Googling) it seems like there is a market gap for a tool that allows for a seamless migration away from a Postgres DB that won't allow for replicas.

I imagine that, given some constraints on what queries you allow on your database, you could proxy connections to a managed Postgres database while feeding them simultaneously to a new database. Starting from a snapshot of the old database one could catch the new database up to a live state of the old database and then swap over all connections to the new one.

Does such a proxy already exist? I'd love to know as I could use it. If not it might be fun to build and a good side-project.

I've always used Postgres. For a medium-sized app that I work on right now it's running great. Were I to seriously need more throughput I'd either shard (no small task I know) or use CockroachDB (which to my understanding is basically Postgres with built-in sharding and no extension support). Throwing away relationships, constraints, unique compound indices and all of the tools I love that Postgres provides just to get schemaless JSON with high write throughput out of the box doesn't sound like a good deal. But so many people have made the decision to go NoSQL so there must be something I'm missing.

NewsBlur was down yesterday evening due to its Mongo database getting attacked by a hacker and held for ransom. It’s restored from backup, but there are privacy implications for anyone who had sensitive private data there. We will likely find out more after the maintainer recovers from a busy night.

There are no good links for this, but it’s being discussed on Hacker News. Since it’s open source, someone described what’s being kept in that database.

(I use NewsBlur, but I don’t think my RSS reading habits are all that sensitive. Others might be in a different situation, though.)

I was asked this question in an interview and I said the downsides would be extra disk space used to store it and more disk writes from the database to maintain the index on updates. I asked a friend and he said the downsides are space used in memory to store the index as well as the fact that an index could take days to build on a large dataset.

What does everyone here think about this question?

Hello everyone!

I've recently got an idea for a Database as a Service I'd like to create. The only issue is - I don't know how to create or host a database!

I've only ever used Mongoose/mLab with Javascript, and a minimal amount of Postgres with Python.

If I'm looking to create a database that will, eventually, be able to store images, songs, and videos, where should I start my homework?

I can create the backend and the frontend with no issue - just stuck on this part here. If it's of any relevance, I most frequently use the MERN stack.