Found this particularly outstanding writeup on HN, about The Guardian's migration from MongoDB to Postgres. Recommended full reading: It's insightful, not too long, well-written and very, very...
Exemplary
Found this particularly outstanding writeup on HN, about The Guardian's migration from MongoDB to Postgres. Recommended full reading: It's insightful, not too long, well-written and very, very real-world.
One of my favourite things is seeing how companies whose core competency isn't tech actually do handle tech. I did a lot of freelancing back in the days and I can immediately recognize some patterns which would frustrate the fuck out of me if I worked there.
One thing readers will note is that they started talking about the migration to AWS at least prior to 2015; it became a priority "after July 2015" and it was completed in April 2018. The other thing to note is that they say they had "2.3M content items", which is a tiny, tiny amount of items for Postgres (where you can easily talk about hundreds of millions of rows per table without talking about things like sharding etc).
So I can definitely feel the pain of a company whose processes do not prioritize tech, and of a tech team that is still doing its best despite that. Still, a two very avoidable red flags / rookie mistakes:
Adding an externality as a core dependency of what you're doing. Namely: Waiting for AWS to implement at-rest encryption for DynamoDB before deciding on a new database backend. This should not have even been considered unless they had an account manager who specifically told them "I promise, it's coming before this date". This probably worked out for the best for them, I imagine they'd have had more trouble with DynamoDB than they'll get with boring old Postgres JSONBs.
Load testing being such a pain and taking so long. I've worked in teams like these. This was likely a case of everyone on the team being out of their depth, but not quite enough to admit it or take a different approach. They wasted weeks! Load testing is a fairly easy thing to do and they shouldn't have been in a situation where it was so dangerous to production when they weren't even testing against production. I agree with their assessment that calling it off was the best move (the best time to do something would have been 3 weeks earlier, the second best time was right there and then). If you're a more tech-oriented startup and you end up there though, you absolutely need to hire someone who can deal with this; it's pretty unacceptable to find yourself in such a situation.
Man, reading that article was painful. Statements like that one really indicate to me that they had no idea what the hell they were doing. And rather than take the time to read up on it and get...
Man, reading that article was painful.
This was non-trivial, as Mongo didn’t provide any tooling for getting set up easily on AWS
Statements like that one really indicate to me that they had no idea what the hell they were doing. And rather than take the time to read up on it and get comfortable with the tech, they waited around for months for someone else to update their tech so they could just use different tech entirely.
Migrating to AWS isn't exactly difficult. If you're already running a replica set, you literally just bring up a new secondary, connect it, let it replicate, and have the secondary become the new primary. Then set up new secondaries on AWS and drop your old ones (or keep one up as a non-promotable backup).
I mean, fuck, don't they read documentation?
That's just one nugget. I would imagine that their code base is just one giant mess of technical debt and that they just throw in whatever functioning code they can while working around it. There's nothing in this article that could possibly make me think otherwise.
They need an engineering lead who actually knows what the fuck they're doing and they need to listen to said lead. They need to allocate budget to getting their shit together. This whole situation has "we only pay for the cheapest developers we can" written all over it.
Like I said in my other post though, tech isn't their core competency. I really don't know that they need a better engineering team. They could potentially save money if they had one, but one...
They need an engineering lead who actually knows what the fuck they're doing and they need to listen to said lead. They need to allocate budget to getting their shit together. This whole situation has "we only pay for the cheapest developers we can" written all over it.
Like I said in my other post though, tech isn't their core competency. I really don't know that they need a better engineering team. They could potentially save money if they had one, but one thing I realized at the previous company I started is that "saving money" isn't necessarily a goal, in and of itself.
A company can have various goals. People like to talk about the bottom line, but more specifically, most companies want to make all their employees and owners money, while giving them fulfilling work.
"Saving money" sounds like it can help reach that goal but actually it usually doesn't, unless the savings are significant and visible in the short term. Because companies more often adapt to their cashflow, than the other way around. So maybe some of the difference will be reflected at some point, down the line, in someone's christmas bonus. Maybe it'll also get reinvested into a project that fails even harder and wastes more money.
So The Guardian could save money if it had a better engineering team. But it could also probably save money 30 other different ways, 20 of which will be things they can much more readily tackle than improving their engineering.
They could have better tech, but do they need better tech? The fact they're getting by just fine when you read a post-mortem like this one is a testament that they don't. The world runs on hacks after all.
I think we as humans see the world like this far too often. We see the flaws in others' lives, because we know better, but we don't see the myriad of other flaws where we don't know better (both in us, and in others). I always think of dentists who say "You should floss more", because I absolutely know I should, yet I refuse to believe it affects my life in a way I actually care or will care (whereas the act of flossing itself would impact my daily life by adding to my routine).
So what's the lesson here? From my reading of the article, I do think you're right and higher quality tech could overall improve The Guardian, by saving them money and expanding their realm of possibility. But I only know what I've read in that article. I don't know anything about all the potential improvements to their daily news sourcing, interviews, processes, security, verification, legal team, accounting, anything.
So what they "need" is most likely someone who knows a lot more about how well the company is doing in all of those areas. Usually that'd be the executive team. In essence, our assertion that they need better tech can be summarized as: "The CEO is undervaluing tech as an asset to the company".
A teammate and I are spearheading a database migration now, and coincidentally these articles have been popping up in my world lately. It's crazy how similar each process is, yet everyone has...
A teammate and I are spearheading a database migration now, and coincidentally these articles have been popping up in my world lately. It's crazy how similar each process is, yet everyone has these duplicated learnings. Perhaps nobody wrote about it for so long that there's a wave of painstaking migrations. As for this instance, leaving that self-killing proxy is unthinkable, even as a temporary solution.
Found this particularly outstanding writeup on HN, about The Guardian's migration from MongoDB to Postgres. Recommended full reading: It's insightful, not too long, well-written and very, very real-world.
One of my favourite things is seeing how companies whose core competency isn't tech actually do handle tech. I did a lot of freelancing back in the days and I can immediately recognize some patterns which would frustrate the fuck out of me if I worked there.
One thing readers will note is that they started talking about the migration to AWS at least prior to 2015; it became a priority "after July 2015" and it was completed in April 2018. The other thing to note is that they say they had "2.3M content items", which is a tiny, tiny amount of items for Postgres (where you can easily talk about hundreds of millions of rows per table without talking about things like sharding etc).
So I can definitely feel the pain of a company whose processes do not prioritize tech, and of a tech team that is still doing its best despite that. Still, a two very avoidable red flags / rookie mistakes:
Adding an externality as a core dependency of what you're doing. Namely: Waiting for AWS to implement at-rest encryption for DynamoDB before deciding on a new database backend. This should not have even been considered unless they had an account manager who specifically told them "I promise, it's coming before this date". This probably worked out for the best for them, I imagine they'd have had more trouble with DynamoDB than they'll get with boring old Postgres JSONBs.
Load testing being such a pain and taking so long. I've worked in teams like these. This was likely a case of everyone on the team being out of their depth, but not quite enough to admit it or take a different approach. They wasted weeks! Load testing is a fairly easy thing to do and they shouldn't have been in a situation where it was so dangerous to production when they weren't even testing against production. I agree with their assessment that calling it off was the best move (the best time to do something would have been 3 weeks earlier, the second best time was right there and then). If you're a more tech-oriented startup and you end up there though, you absolutely need to hire someone who can deal with this; it's pretty unacceptable to find yourself in such a situation.
Man, reading that article was painful.
Statements like that one really indicate to me that they had no idea what the hell they were doing. And rather than take the time to read up on it and get comfortable with the tech, they waited around for months for someone else to update their tech so they could just use different tech entirely.
Migrating to AWS isn't exactly difficult. If you're already running a replica set, you literally just bring up a new secondary, connect it, let it replicate, and have the secondary become the new primary. Then set up new secondaries on AWS and drop your old ones (or keep one up as a non-promotable backup).
I mean, fuck, don't they read documentation?
That's just one nugget. I would imagine that their code base is just one giant mess of technical debt and that they just throw in whatever functioning code they can while working around it. There's nothing in this article that could possibly make me think otherwise.
They need an engineering lead who actually knows what the fuck they're doing and they need to listen to said lead. They need to allocate budget to getting their shit together. This whole situation has "we only pay for the cheapest developers we can" written all over it.
Like I said in my other post though, tech isn't their core competency. I really don't know that they need a better engineering team. They could potentially save money if they had one, but one thing I realized at the previous company I started is that "saving money" isn't necessarily a goal, in and of itself.
A company can have various goals. People like to talk about the bottom line, but more specifically, most companies want to make all their employees and owners money, while giving them fulfilling work.
"Saving money" sounds like it can help reach that goal but actually it usually doesn't, unless the savings are significant and visible in the short term. Because companies more often adapt to their cashflow, than the other way around. So maybe some of the difference will be reflected at some point, down the line, in someone's christmas bonus. Maybe it'll also get reinvested into a project that fails even harder and wastes more money.
So The Guardian could save money if it had a better engineering team. But it could also probably save money 30 other different ways, 20 of which will be things they can much more readily tackle than improving their engineering.
They could have better tech, but do they need better tech? The fact they're getting by just fine when you read a post-mortem like this one is a testament that they don't. The world runs on hacks after all.
I think we as humans see the world like this far too often. We see the flaws in others' lives, because we know better, but we don't see the myriad of other flaws where we don't know better (both in us, and in others). I always think of dentists who say "You should floss more", because I absolutely know I should, yet I refuse to believe it affects my life in a way I actually care or will care (whereas the act of flossing itself would impact my daily life by adding to my routine).
So what's the lesson here? From my reading of the article, I do think you're right and higher quality tech could overall improve The Guardian, by saving them money and expanding their realm of possibility. But I only know what I've read in that article. I don't know anything about all the potential improvements to their daily news sourcing, interviews, processes, security, verification, legal team, accounting, anything.
So what they "need" is most likely someone who knows a lot more about how well the company is doing in all of those areas. Usually that'd be the executive team. In essence, our assertion that they need better tech can be summarized as: "The CEO is undervaluing tech as an asset to the company".
We're engineers, it's no wonder we think that :)
A teammate and I are spearheading a database migration now, and coincidentally these articles have been popping up in my world lately. It's crazy how similar each process is, yet everyone has these duplicated learnings. Perhaps nobody wrote about it for so long that there's a wave of painstaking migrations. As for this instance, leaving that self-killing proxy is unthinkable, even as a temporary solution.