There is a fairly new ASIC named "Sohu" that is purpose-built for transformers. They have some bold claims that are impressive if true.

I found a short discussion[2] you may find useful.

[1]: https://www.etched.com/

[2]: https://www.lesswrong.com/posts/qhpB9NjcCHjdNDsMG/new-fast-t...

This is just a PowerPoint slide at this point.

These numbers are just the hole from GPUs that have already been bought/ordered. Today's GPUs will inevitably be replaced by something, whether it be better GPUs, NPUs/TPUs, ASICs, or FPGAs. As chips get cheaper in the future the hole will grow at a slower rate but it only grows.

>Is there an intrinsic reason something similar couldn't happen with LLMs?

Only if we can increase the efficiency of LLMs by 2-3 orders of magnitude, there are only some in lab examples of this and nothing really being publicly shown.

Even then the models are still going to require rather large amounts of memory, and any performance increases that could boost model efficiency would very likely increase performance on GPU hardware to the point we could get continuous learning models from multimodal input like video data and other sensors.

There is a reason why we don't use ASICs and instead use GPUs.

While people may say something is a Transformer that's more of a general description. It's not a specific algorithm; there are countless transformers and people are making progress on finding new ones.

Bitcoin runs a specific algorithm that never changes. That's for an ASIC. AI/ML runs a large class of models. GPUs are already finely tuned for his case.

Nvidia and AMD bought out the big FPGA makers.

AMD bought Xilinx, but Intel recently spun off Altera.

Opps, yeah was thinking of Intel's Altera buy, not Nvidia, which I guess is now undone.