Power consumption of LLM's
I haven't been closely following the releases of new models and the research papers that sometimes go along with them. So I was wondering: is power consumption ever seriously talked about by OpenAI, Google, Anthropic, and others? Do we get some specific numbers of how much power their models actually consume to produce 100 tokens? The cost of training some of these models? Or we're not yet at that stage yet and nobody cares and at best we can just get really rough estimates based on "trust me bro" tweets by their respective CEO's?
Some time ago, I came across this post. GPT-OSS (2OB, 120B) are meager, yet energy efficient models and I was surprised that the power consumption is still larger than what I had estimated before.
For 120B to generate a 1000 tokens (which is really not a lot) it would take up around 83 Wh of electricity. For context my PC consumes ~100Wh when idling. So it's almost equivalent as leaving my PC on for an hour. Considering that proprietary models are TRILLIONS of parameters and probably not as energy efficient the true power consumption of these models is concerning. Of course, these big data centers do a lot of things to maximize the efficiency that this "test" fails to do. But even if you half the energy consumption it's still significant given their size and their pervasiveness in handling everyday, trivial tasks.
I haven't come across similar posts or studies for newer open source models, so if somebody has, please share.
Also, I don't seem to fully understand how does context size fit into all of this. The larger the context size the more power it would take to produce those 100 tokens?
I don't think they publish any official numbers, not sure. I remember this post from a while ago that said:
https://simonpcouch.com/blog/2026-01-20-cc-impact/
But people pointed out that, back then, if you included the model training costs as well it would be more like running 10 extra fridges. Hard to tell how that compare to today's numbers.
I think the current costs of LLM inference is "not too bad", at least for Open Source models you can run at home. It's probably much higher for the larger models due to the hardware requirements.
But the biggest environment damage comes from model training and, even worse, the new data centers being built. The energy consumption, water, the environmental consequences, all the electronic waste from hardware that only last a few years, etc.