> we … defined emergent abilities as “abilities that are not present in small language models but are present in large language models.”
This is a poor definition because it doesn’t match what is generally meant by emergent behavior and abilities, which leads to people talking past each other.
For example, it can easily be true that LLMs have emergent behaviors by this definition and not be greater than the sum of their parts.
A better definition would be exhibiting abilities or behaviors not in the training data.
Hmm; I would think a better definition would include a description of the behavior as being greater than the sum of its parts. I can imagine abilities and behaviors being in the training data but not exhibited by a smaller LLM but being present in a bigger one. My favorite example is the ability to multiply large number consistently and accurately (e.g. 23540932 x 3274, which ChatGPT-4 just failed to do correctly for me when I asked it). A better LLM would learn how to carry the process out step-by-step.
This is a poor definition because it doesn’t match what is generally meant by emergent behavior and abilities, which leads to people talking past each other.
For example, it can easily be true that LLMs have emergent behaviors by this definition and not be greater than the sum of their parts.
A better definition would be exhibiting abilities or behaviors not in the training data.