Cambrian explosion implies that there’s a huge variety of different creatures out there, but I suspect those bots are all just wrappers around OpenAI/anthropic models.
This is more like the rise of Cyanobacteria as a single early dominant lifeform
Writing a crawler that's a wrapper around OpenAI or Anthropic doesn't make sense to me: what is your crawler doing? Piping all that crawler data through an existing LLM would cost you millions of dollars, and for what purpose?
Crawling to train your own LLM from scratch makes a lot more sense.
This is more like the rise of Cyanobacteria as a single early dominant lifeform