The context comes from the attention mechanism, not from word embeddings. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		eggdaft on May 6, 2024 \| parent \| context \| favorite \| on: How LLMs Work, Explained Without Math The context comes from the attention mechanism, not from word embeddings.

mjburgess on May 6, 2024 [–]

Run attention on an ordinal word embedding and see what happens

eggdaft on May 6, 2024 | [–]

Well yes, necessary but not sufficient, obviously.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact