And, the attention mechanism scales quadratically with context length. This is where all of the insane memory bandwidth requirements come from.