I agree this is the main case where it makes sense.
But a recent trend that cut into the cost savings is that foundation model companies have started releasing small models. So you can build a use case with qwen 235B, then shrink down to 30B, or even all the way down to 0.6B if you really want to.
The smaller models lose some accuracy, but some use cases are solvable even by these smaller and much more efficient models.
But a recent trend that cut into the cost savings is that foundation model companies have started releasing small models. So you can build a use case with qwen 235B, then shrink down to 30B, or even all the way down to 0.6B if you really want to.
The smaller models lose some accuracy, but some use cases are solvable even by these smaller and much more efficient models.