Google announced they are serving 500T tokens per month.
State of the art models are currently trained with less than 30T tokens. Even if training tokens are more costly to run (eg, a factor of 3x for forward, backward, and weight updates, and take another factor of 2x for missing quantization), you end up in a situation where inference compute dominates training after a very short time of amortization.
This is a good point. Another point is that the better models get, the less wasted tokens there will be on unproductive token generation for answers that are wrong in some way. Better answers might lead to increased demand of course. But less waste is not a bad thing in itself. And improved quality of the answers has other economical advantages.
My view is that increased energy demand is not necessarily a bad thing in itself. First, it's by no means the dominant source of such demand, other things (transport, shipping, heating, etc.) outrank it; so a little bit of pressure from AI won't move the needle too much. Our main problem remains the same: too much CO2 being emitted. Second, meeting increased demand is typically done with renewables these days. Not because it's nice to do so but because it's cheap to do so. That's why renewables are popular in places like Texas. They don't care about the planet there. But they love cheap energy. And the more cheap, clean power we bring online, the worse expensive dirty power actually looks.
Increased demand leads to mostly new clean generation and increased pressure to deprecate dirty expensive generation. That's why coal is all but gone from most energy markets. That has nothing to do with how dirty it is and everything to do with how expensive it is. Gas based generation is heading the same direction. Any investment in such generation should be considered as very risky.
Short term of course you get some weird behavior like data centers being powered by gas turbines. Not because it's cheap but because it's easy and quick. Long term, a cost optimization would be getting rid of the gas generators. And with inference increasingly becoming the main thing in terms of energy and tokens, energy also becomes the main differentiator for profitability of AI services. Which again points at using cheap renewables to maximize profit. The winners in this market will be working on efficiency. And part of that is energy efficiency. Because that and the hardware is the main cost involved here.
Depends. For CoT models inference is significantly more costly (compared to regular models).
Also,
>Brent Thill of Jefferies, an analyst, estimates that [inference] accounts for 96% of the overall energy consumed in data centres used by the AI industry.
Foreword Author here.
I agree, even early estimates e.g. from Meta (2022) suggested 20% Training, 10% Experiments, 70% inference. And adoption is rising from month to month.
Doesn't training the model consume the most energy in most cases?