So nearly two-thirds of their monthly budget is going to hosting fees. And if you dig into that linked post...oof. They've completely over-engineered it and are spending a ton of unnecessary money...
So nearly two-thirds of their monthly budget is going to hosting fees. And if you dig into that linked post...oof. They've completely over-engineered it and are spending a ton of unnecessary money on Azure.
First, here are the technology goals we started with when building the platform:
It should be able to handle 500,000 registered users with 50,000 (10%) simultaneous active users
Should the platform ever need to grow, it should be architected in such a way that a complete rewrite isn't necessary. With this in mind, we didn't set out to architect a solution that could handle 100 million users. We tried to find a comfortable middle ground
This is an extremely common pitfall for a lot of people when building a webapp. "Make it infinitely scalable so that when we inevitably hit the big time we won't need to do anything other than twist a dial with our hosting provider".
From some of the replies to that post:
It should be able to handle 500,000 registered users with 50,000 (10%) simultaneous active users
And how's that working out? It looks insanely ambitious to me! Has there ever even been as many as 50 simultaneous users?
Yeah it turned out to be a bit optimistic :) The most traffic we had at once was when we were featured on the front page of the BBC, and there were a few thousand people browsing. We had to scale up briefly but everything ran smooth.
"We had to scale up briefly" - meaning that despite claiming "architected for 50k simultaneous users", they hit a few thousand active users (presumably most or all not logged in, so they could be served static / cached content) and they had to scale up to meet that demand. Meaning that if they actually had that sustained level of users, they'd need even more than $900/mo in hosting fees.
It makes no sense to be paying that much in hosting. You can serve millions of (dynamic, database driven) daily requests on a single off the shelf server with some extra RAM. You can easily scale...
It makes no sense to be paying that much in hosting. You can serve millions of (dynamic, database driven) daily requests on a single off the shelf server with some extra RAM. You can easily scale up from there by adding new machines. They shouldn't be paying more than a couple hundred a month, and could easily go even cheaper.
Or if you want turnkey scalability you can serve far more traffic than they likely have with any number of $50-100/month cloud solutions and dial up the capacity if needed.
Note that I have no idea what their backend looks like, it could be such a mess that they have to throw money at the problem.
In any case this seems like a great example of what happens when there's no one in the room with comprehensive, real world experience in web apps.
A quick search on subreddit status shows the sub gets just 45 posts and 962 comments per day with total subscriber count of 1,130,222. I've joined too but only visit if I see a trending post in my...
A quick search on subreddit status shows the sub gets just 45 posts and 962 comments per day with total subscriber count of 1,130,222. I've joined too but only visit if I see a trending post in my feed. As I'm writing this comment their sidebar shows just 4473 users online on the sub. Total subscriber count is quite misleading imo which is the only thing they took into consideration I think. In comparison India sub has 3940 users online with just 300k subscribers.
That doesn't really change the hosting costs. The whole point of AWS and azure (or, well, part of the point) is that they're pay as you go. You pay for how much resources you use. They may aim for...
That doesn't really change the hosting costs. The whole point of AWS and azure (or, well, part of the point) is that they're pay as you go. You pay for how much resources you use.
They may aim for 500,000 users, but they're paying, hosting wise, for however much resources their users are actually using.
Well...yes and no. They're paying for the capacity they've allocated with Azure, whether they're using it or not to actively serve users. If you provision 10 VMs, you're going to pay $X/month for...
Well...yes and no. They're paying for the capacity they've allocated with Azure, whether they're using it or not to actively serve users. If you provision 10 VMs, you're going to pay $X/month for them whether they're sitting at 10% CPU usage or 90%.
They're also using a lot of "value-added" type Azure services that are not just hosted VMs (whatever Azure calls their EC2 clone) but the more managed services like App Service, Service Bus, Azure Cache, Azure Search, etc. Each one of those is charged piecemeal, which adds up.
I hadn't really looked at Azure's pricing model before, but...holy crap. This is worse than I thought.
Their App Service pricing on Linux for example, the "standard plan" is $0.095/hour (or $68/month). And that gets you...a VM with 1 core, 1.75 GB RAM, and 50 GB of disk.
(and the "premium" service plan mentions SSD storage as one of the selling points, which leads me to believe the standard plan runs on spinning rust?!?)
The hosting provider I use, Vultr, will charge you $24/mo for a better-spec'd VM (2 cores, 4 GB RAM, 128GB SSD). Obviously this is apples-to-oranges because that's just a VM and not an "app service", but it gives you an idea how much of a markup Azure is charging for their value-added-ness.
They're also doing wasteful things like running two SQL instances, one for IdentityServer and one for their own purposes. I really doubt that they need both instances when two databases on a single instance would work fine. And this isn't very clear from that post, but it sounds like they're also running two separate App Service instances, one for each, rather than multiple services on a single VM (which I sure hope Azure allows you to do?)
Oh, and all of this is duplicated between dev and production. I'm all for testing somewhere other than prod, but that possibly doubles the hosting bill (I would hope they're at least running smaller/cheaper instances for their dev stack).
So yes, they only pay for what they use, but the "bare minimum" of what they need to use in order to run the site for 1 user a minute is very high. They've hit an antipattern I've often seen with use of the cloud, which is they focus too much on "scaling up" but never pay attention to scaling down. Great, your app can handle 1 million requests/second when you deploy it to AWS and sign over your first-born to Jeff Bezos. Can I run it on my laptop if I only need it to do 1 rps? Crucially, can I run it on my laptop and have it coexist nicely with a bunch of Chrome tabs, an text editor / IDE, etc?
You could pretty easily run that entire site on the $24/mo VM I linked above. Run Postgres with full-text search, Redis for caching (btw, they also have two caching layers - whhhhhhhhhy?) and either Redis or RabbitMQ for the "service bus". Azure Functions for scheduled jobs can be replaced with cron.
Doing all of that would require a significant amount more time from the operator of the website than just "throw a bunch of Azure services at it". But, as a startup / side gig type website without paying customers, time is theoretically something the owner has more of than money. The goal he seems to have pursued is "use enough Azure services that I don't have to do any maintenance work" and the end result is he pays Azure the equivalent of a part-time low-wage worker ($900/mo = 20 hours/week at $11.25/hour).
Whaat? My 5$/mo VPS from <basically everywhere I tried, but currently Linode> gets 1 core, 1GB RAM, 20GB disk. 5 dollars! What is going on with Azure that it justifies that much mark-up?!? I...
Their App Service pricing on Linux for example, the "standard plan" is $0.095/hour (or $68/month). And that gets you...a VM with 1 core, 1.75 GB RAM, and 50 GB of disk.
Whaat? My 5$/mo VPS from <basically everywhere I tried, but currently Linode> gets 1 core, 1GB RAM, 20GB disk. 5 dollars! What is going on with Azure that it justifies that much mark-up?!?
They've hit an antipattern I've often seen with use of the cloud, which is they focus too much on "scaling up" but never pay attention to scaling down. Great, your app can handle 1 million requests/second when you deploy it to AWS and sign over your first-born to Jeff Bezos. Can I run it on my laptop if I only need it to do 1 rps?
I agree, and more so if you're planning on letting people to self-host the damn thing. Why would GitLab require 4 cores and 4 GB's of RAM?? Why does Mastodon transpiling it's JS and CSS eat all my RAM, and then some!?
(Also, pretty please make your thing work without SMTP access even if it means no e-mail notifications are sent. I don't care about them)
I'm not making this post to try to encourage anyone to pledge (though if you're interested in the project, please do support it), but more because I think Ceasefire (previously named ChangeAView)...
I'm not making this post to try to encourage anyone to pledge (though if you're interested in the project, please do support it), but more because I think Ceasefire (previously named ChangeAView) is going to serve as a prime example of how difficult it is for even a prominent Reddit community to split off into a stand-alone site.
I think they've certainly made some mistakes in their approach (for example, no longer being able to browse content on the site without registering first), but overall it's never really managed to get much activity, despite getting a decent amount of coverage in some prominent media outlets (e.g. the BBC).
I'm sure a lot of people will want to criticize their operational costs shown in the post as being much too high for the site's size—and they are—but that's really not the core problem. $1500/month isn't what's making the difference between it being a viable business or not, and wouldn't even cover the cost of a single minimum-wage employee for most businesses.
I think part of the effectiveness of some of the high quality subs, like AskHistorians or AskScience, is that actual subject matter experts used to just hang out on Reddit and screw around so you...
serve as a prime example of how difficult it is for even a prominent Reddit community to split off into a stand-alone site.
I think part of the effectiveness of some of the high quality subs, like AskHistorians or AskScience, is that actual subject matter experts used to just hang out on Reddit and screw around so you had a decent shot at getting the attention of someone whose opinion on the subject was worth a damn. Without that there's no point.
If you have a stand alone "Ask Historians" site, any historian there would have nothing to do but write up long, unpaid answers to questions. They wouldn't even really care about asking history questions in it, because if they're academics they could just email someone they know who is an actual expert on the topic if they're actually curious. There's nothing in it for them but being an unpaid tutor. A Historian on Reddit, on the other hand, is doing all the random reddit stuff and occasionally popping in on a history question because they happen to have the time.
I just don't see how you can make things like that standalone. Not enough people would care enough to check-in on it. And I see this on Reddit now. I think they've started to bleed a lot of its older user base and very very few of the highly voted questions on AskScience or AskHistorians get answered anymore.
Do you know much about what motivates them to participate? Is my suspicion right or do they just enjoy educating the masses on Biblical exegesis and would do it on a dedicated forum?
Do you know much about what motivates them to participate? Is my suspicion right or do they just enjoy educating the masses on Biblical exegesis and would do it on a dedicated forum?
I guess it depends what exactly you mean by "success", but I'd say the biggest thing overall is just being careful to avoid any kind of "outside influences" like investors or certain business...
I guess it depends what exactly you mean by "success", but I'd say the biggest thing overall is just being careful to avoid any kind of "outside influences" like investors or certain business models (or more accurately for Tildes, all business models).
The costs to keep Tildes running are very low, and in the end any decision related to shutting the site down would be an entirely voluntary one made by me (excepting some kind of crazy legal issue or something). There's no possibility of "we can't afford to keep running the site because our major advertiser pulled out", or "the investors want to terminate the business", or any of the other reasons sites end up having to shut down.
I can run Tildes for as long as I want, however I want. By "I" there, I actually mean more like "we"—the users' opinions are important in decisions as well, but my point is that there's absolutely no external pressure to do anything with the site that I/we don't want to. That's incredibly freeing to know.
I don't want to insult ceasefire's design choices, but there's a reason Azure has pretty much zero non enterprise presence on the web. Their startup costs to bootstrap something small can easily...
I don't want to insult ceasefire's design choices, but there's a reason Azure has pretty much zero non enterprise presence on the web. Their startup costs to bootstrap something small can easily be 100x that of GCP or AWS. Seems like every few months on Hacker News there's also a similar horror story.
I think we as cloud developers need to start aggressively informing people, in every guide or tutorial we write, that even if a solution is platform agnostic, you should really avoid Azure (particularly Azure VMs) unless you actually have the capital to justify it. I'm actually working on a tutorial for college students right now, definitely throwing this in there.
also, a friendly reminder: You may not actually need a server, virtual or otherwise, and if you architect your application without one, you can often cut your costs by a factor of five to ten.
I'll try to remember to post it to tildes once I wrap it up. I'm actual doing it on behalf of a local community college that my non-profit is partnering with, but the info will be generally...
I'll try to remember to post it to tildes once I wrap it up. I'm actual doing it on behalf of a local community college that my non-profit is partnering with, but the info will be generally applicable (insofar as routing web requests via geofencing is generally applicable).
So nearly two-thirds of their monthly budget is going to hosting fees. And if you dig into that linked post...oof. They've completely over-engineered it and are spending a ton of unnecessary money on Azure.
This is an extremely common pitfall for a lot of people when building a webapp. "Make it infinitely scalable so that when we inevitably hit the big time we won't need to do anything other than twist a dial with our hosting provider".
From some of the replies to that post:
"We had to scale up briefly" - meaning that despite claiming "architected for 50k simultaneous users", they hit a few thousand active users (presumably most or all not logged in, so they could be served static / cached content) and they had to scale up to meet that demand. Meaning that if they actually had that sustained level of users, they'd need even more than $900/mo in hosting fees.
It makes no sense to be paying that much in hosting. You can serve millions of (dynamic, database driven) daily requests on a single off the shelf server with some extra RAM. You can easily scale up from there by adding new machines. They shouldn't be paying more than a couple hundred a month, and could easily go even cheaper.
Or if you want turnkey scalability you can serve far more traffic than they likely have with any number of $50-100/month cloud solutions and dial up the capacity if needed.
Note that I have no idea what their backend looks like, it could be such a mess that they have to throw money at the problem.
In any case this seems like a great example of what happens when there's no one in the room with comprehensive, real world experience in web apps.
Wow, they're spending more per month on this project that my company does on the big legacy Rails app I develop for.
A quick search on subreddit status shows the sub gets just 45 posts and 962 comments per day with total subscriber count of 1,130,222. I've joined too but only visit if I see a trending post in my feed. As I'm writing this comment their sidebar shows just 4473 users online on the sub. Total subscriber count is quite misleading imo which is the only thing they took into consideration I think. In comparison India sub has 3940 users online with just 300k subscribers.
They made very poor decision in growth aspects.
That doesn't really change the hosting costs. The whole point of AWS and azure (or, well, part of the point) is that they're pay as you go. You pay for how much resources you use.
They may aim for 500,000 users, but they're paying, hosting wise, for however much resources their users are actually using.
Well...yes and no. They're paying for the capacity they've allocated with Azure, whether they're using it or not to actively serve users. If you provision 10 VMs, you're going to pay $X/month for them whether they're sitting at 10% CPU usage or 90%.
They're also using a lot of "value-added" type Azure services that are not just hosted VMs (whatever Azure calls their EC2 clone) but the more managed services like App Service, Service Bus, Azure Cache, Azure Search, etc. Each one of those is charged piecemeal, which adds up.
I hadn't really looked at Azure's pricing model before, but...holy crap. This is worse than I thought.
Their App Service pricing on Linux for example, the "standard plan" is $0.095/hour (or $68/month). And that gets you...a VM with 1 core, 1.75 GB RAM, and 50 GB of disk.
(and the "premium" service plan mentions SSD storage as one of the selling points, which leads me to believe the standard plan runs on spinning rust?!?)
The hosting provider I use, Vultr, will charge you $24/mo for a better-spec'd VM (2 cores, 4 GB RAM, 128GB SSD). Obviously this is apples-to-oranges because that's just a VM and not an "app service", but it gives you an idea how much of a markup Azure is charging for their value-added-ness.
They're also doing wasteful things like running two SQL instances, one for IdentityServer and one for their own purposes. I really doubt that they need both instances when two databases on a single instance would work fine. And this isn't very clear from that post, but it sounds like they're also running two separate App Service instances, one for each, rather than multiple services on a single VM (which I sure hope Azure allows you to do?)
Oh, and all of this is duplicated between dev and production. I'm all for testing somewhere other than prod, but that possibly doubles the hosting bill (I would hope they're at least running smaller/cheaper instances for their dev stack).
So yes, they only pay for what they use, but the "bare minimum" of what they need to use in order to run the site for 1 user a minute is very high. They've hit an antipattern I've often seen with use of the cloud, which is they focus too much on "scaling up" but never pay attention to scaling down. Great, your app can handle 1 million requests/second when you deploy it to AWS and sign over your first-born to Jeff Bezos. Can I run it on my laptop if I only need it to do 1 rps? Crucially, can I run it on my laptop and have it coexist nicely with a bunch of Chrome tabs, an text editor / IDE, etc?
You could pretty easily run that entire site on the $24/mo VM I linked above. Run Postgres with full-text search, Redis for caching (btw, they also have two caching layers - whhhhhhhhhy?) and either Redis or RabbitMQ for the "service bus". Azure Functions for scheduled jobs can be replaced with cron.
Doing all of that would require a significant amount more time from the operator of the website than just "throw a bunch of Azure services at it". But, as a startup / side gig type website without paying customers, time is theoretically something the owner has more of than money. The goal he seems to have pursued is "use enough Azure services that I don't have to do any maintenance work" and the end result is he pays Azure the equivalent of a part-time low-wage worker ($900/mo = 20 hours/week at $11.25/hour).
Whaat? My 5$/mo VPS from <basically everywhere I tried, but currently Linode> gets 1 core, 1GB RAM, 20GB disk. 5 dollars! What is going on with Azure that it justifies that much mark-up?!?
I agree, and more so if you're planning on letting people to self-host the damn thing. Why would GitLab require 4 cores and 4 GB's of RAM?? Why does Mastodon transpiling it's JS and CSS eat all my RAM, and then some!?
(Also, pretty please make your thing work without SMTP access even if it means no e-mail notifications are sent. I don't care about them)
I'm not making this post to try to encourage anyone to pledge (though if you're interested in the project, please do support it), but more because I think Ceasefire (previously named ChangeAView) is going to serve as a prime example of how difficult it is for even a prominent Reddit community to split off into a stand-alone site.
I think they've certainly made some mistakes in their approach (for example, no longer being able to browse content on the site without registering first), but overall it's never really managed to get much activity, despite getting a decent amount of coverage in some prominent media outlets (e.g. the BBC).
I'm sure a lot of people will want to criticize their operational costs shown in the post as being much too high for the site's size—and they are—but that's really not the core problem. $1500/month isn't what's making the difference between it being a viable business or not, and wouldn't even cover the cost of a single minimum-wage employee for most businesses.
I think part of the effectiveness of some of the high quality subs, like AskHistorians or AskScience, is that actual subject matter experts used to just hang out on Reddit and screw around so you had a decent shot at getting the attention of someone whose opinion on the subject was worth a damn. Without that there's no point.
If you have a stand alone "Ask Historians" site, any historian there would have nothing to do but write up long, unpaid answers to questions. They wouldn't even really care about asking history questions in it, because if they're academics they could just email someone they know who is an actual expert on the topic if they're actually curious. There's nothing in it for them but being an unpaid tutor. A Historian on Reddit, on the other hand, is doing all the random reddit stuff and occasionally popping in on a history question because they happen to have the time.
I just don't see how you can make things like that standalone. Not enough people would care enough to check-in on it. And I see this on Reddit now. I think they've started to bleed a lot of its older user base and very very few of the highly voted questions on AskScience or AskHistorians get answered anymore.
Do you know much about what motivates them to participate? Is my suspicion right or do they just enjoy educating the masses on Biblical exegesis and would do it on a dedicated forum?
I guess it depends what exactly you mean by "success", but I'd say the biggest thing overall is just being careful to avoid any kind of "outside influences" like investors or certain business models (or more accurately for Tildes, all business models).
The costs to keep Tildes running are very low, and in the end any decision related to shutting the site down would be an entirely voluntary one made by me (excepting some kind of crazy legal issue or something). There's no possibility of "we can't afford to keep running the site because our major advertiser pulled out", or "the investors want to terminate the business", or any of the other reasons sites end up having to shut down.
I can run Tildes for as long as I want, however I want. By "I" there, I actually mean more like "we"—the users' opinions are important in decisions as well, but my point is that there's absolutely no external pressure to do anything with the site that I/we don't want to. That's incredibly freeing to know.
I don't want to insult ceasefire's design choices, but there's a reason Azure has pretty much zero non enterprise presence on the web. Their startup costs to bootstrap something small can easily be 100x that of GCP or AWS. Seems like every few months on Hacker News there's also a similar horror story.
I think we as cloud developers need to start aggressively informing people, in every guide or tutorial we write, that even if a solution is platform agnostic, you should really avoid Azure (particularly Azure VMs) unless you actually have the capital to justify it. I'm actually working on a tutorial for college students right now, definitely throwing this in there.
also, a friendly reminder: You may not actually need a server, virtual or otherwise, and if you architect your application without one, you can often cut your costs by a factor of five to ten.
As a CS student, I’d be very interested in the article. All I’ve used is DigitalOcean and AWS Lightsail - with DO being my preferred.
I'll try to remember to post it to tildes once I wrap it up. I'm actual doing it on behalf of a local community college that my non-profit is partnering with, but the info will be generally applicable (insofar as routing web requests via geofencing is generally applicable).