Writing a crawler that's a wrapper around OpenAI or Anthropic doesn't make sense to me: what is your crawler doing? Piping all that crawler data through an existing LLM would cost you millions of dollars, and for what purpose?

Crawling to train your own LLM from scratch makes a lot more sense.