Power consumption of LLM's
I haven't been closely following the releases of new models and the research papers that sometimes go along with them. So I was wondering: is power consumption ever seriously talked about by OpenAI, Google, Anthropic, and others? Do we get some specific numbers of how much power their models actually consume to produce 100 tokens? The cost of training some of these models? Or we're not yet at that stage yet and nobody cares and at best we can just get really rough estimates based on "trust me bro" tweets by their respective CEO's?
Some time ago, I came across this post. GPT-OSS (2OB, 120B) are meager, yet energy efficient models and I was surprised that the power consumption is still larger than what I had estimated before.
For 120B to generate a 1000 tokens (which is really not a lot) it would take up around 83 Wh of electricity. For context my PC consumes ~100Wh when idling. So it's almost equivalent as leaving my PC on for an hour. Considering that proprietary models are TRILLIONS of parameters and probably not as energy efficient the true power consumption of these models is concerning. Of course, these big data centers do a lot of things to maximize the efficiency that this "test" fails to do. But even if you half the energy consumption it's still significant given their size and their pervasiveness in handling everyday, trivial tasks.
I haven't come across similar posts or studies for newer open source models, so if somebody has, please share.
Also, I don't seem to fully understand how does context size fit into all of this. The larger the context size the more power it would take to produce those 100 tokens?
I don't think they publish any official numbers, not sure. I remember this post from a while ago that said:
https://simonpcouch.com/blog/2026-01-20-cc-impact/
But people pointed out that, back then, if you included the model training costs as well it would be more like running 10 extra fridges. Hard to tell how that compare to today's numbers.
I think the current costs of LLM inference is "not too bad", at least for Open Source models you can run at home. It's probably much higher for the larger models due to the hardware requirements.
But the biggest environment damage comes from model training and, even worse, the new data centers being built. The energy consumption, water, the environmental consequences, all the electronic waste from hardware that only last a few years, etc.
And bear in mind model training costs never go down unless they stop training models entirely.
The whole point is that they're maximizing available compute.
Indeed. I feel that if we could just stop now with pre training and focus on post training and better harness and other tooling, things would keep improving and the damage would be way more acceptable moving forward. But capitalism will never let that happen.
The scaling from 120B to 1T+ parameters likely isn’t as bad as it might look if you just extrapolate from that post, at least based on my understanding of their methodology. They’re using the same 8x A100 machine for all tests - sensibly so, to get a like for like comparison - but that’s an older machine and kinda overkill for the model sizes they’re testing, so I’d guess that it’s a good baseline for comparing the models tested but less so for getting an absolute W/token value that applies to bigger models on newer GPUs with the batching and VRAM allocations tuned perfectly.
Since we don't know what frontier model power use looks like, we can extrapolate from open weight model power use and just scale it up to larger models. It's not perfect but even with a very large margin of error it's clear that the power consumption from individual LLM use is negligible.
That includes training because, to do the math in good faith, you need to amortize the training cost among all users over the full life of the model.
Without doing emotionally motivated math I don't see a way to make individual usage into a significant concern.
The concern is about the power demands of the industry as a whole. How much will it increase prices? How many coal and gas plants will stay online longer (as opposed to being replaced by renewables) to meet demand? Will legislators do anything about it? If datacenters had to factor in the cost of their impact and otherwise take responsibility for externalities, it would solve most of the problems. They'd be investing in renewable energy for their facilities, installing closed loop cooling (it's not new technology, just a bit more expensive) and taking efficiency seriously.
As it is right now in many places, datacenters are able to offload costs onto the surrounding communities via the power grid and water supply. The solution is to make them pay those costs.
IMO it's a mistake to follow the petrocarbon industry climate change playbook and misattribute responsibility onto individuals to distract from the only real problem: The datacenter operators, the model labs and the money behind them.
I've attempted to calculate the energy cost of LLM last year.
I don't remember the exact numbers, but I ended up with about 2 to 8 kWh per user per year (about the same as running an oven to cook 5 cakes). I used to think that's not so bad, but it's likely underestimated, and seems like it's going to grow a lot more before it stops.
I calculated that by calculating how much electricity is consummed by all AI specialised GPU (since barely anyone else than AI datacentre buy significant amount, and I assume they run them close to 24/7 either for inference or for training) and ignored the consumption of other datacenter's component (I probably shouldn't).
and then divided that by the number of ChatGPT user (since I expect all user of AI to have had a chatGPT account at one point). I'd like to have a better metrics because I expect a lot of those chatGPT account are just people trying it out as a one of for fun, so not real regular users.
Then again, this lumps together image generation and text generation, which aren't really equivalent.
This paper from JP Morgan's energy investment arm was linked in Adam Tooze's Substack letter today, and it's got general answers to the question in terms of data center power utilization and LLM efficiency (not just joules per token, but FLOPs per watt and relative trends):
https://cdn.jpmorganfunds.com/content/dam/jpm-am-aem/global/en/insights/eye-on-the-market/fighting-words-amv.pdf
The key conclusion is that AI chip efficiency improvements are not keeping pace with model scale and utilization demands. There's definitive evidence that in the U.S. at least, power generation and grid delivery isn't capable of supplying the existing and planned data center buildouts, and there's measurable impact on household power prices.
The paper also covers some interesting data on various nations' continuing economic dependencies on fossil fuels and vulnerability to the U.S.-Iran war price shocks. There's just a very broad and deep range of coverage, worth reading all the way through.
And even this is something that can be improved upon, if only there were incentives. My Mac (M3 Max MacBook Pro) is only capable of drawing 140W running full tilt, and it's capable of running some LLMs (or some impressive 3D rendering workloads). At idle, it's under 7W. Even the current M4 Ultra Mac Studio desktop idles around 12W (though it can draw close to 500W in heavy use).
It's wild that the idle draw of commodity desktops, both due to components and poor software management, is so high now. Since the 90s, we've went from CPUs not even needing active coolers and having a single rear case fan to monsters with 1000W PSUs.
It's all probably a drop in the bucket compared to data center power use, since those machines are kept cranked most of the time, as idle hardware is wasted money...but I often think that there's a lot of waste there. The current LLM gold rush isn't even incentivizing finding ways to spend less on electricity, because they all have blank checks and just spin up dirty gas generators and coal plants to feed their excesses.
I don't have an answer but why not use ChatGPT to get an answer.
Modern LLMs that are trained with reasoning and tool calling loops are really good at generating answers from information that is available to it. The problem is that the information is not published by the api providers.
GPT 5.5 High seems to agree:
Thinking and tool call loops use tokens that aren't visible to the user, which changes the answer dramatically. And its reasoning doesn't consider training power usage.