An LLM trained on all other science before Copernicus or Galileo would be expected to explain as true that the world is the flat center of the universe.
This. If we (without AI, even) extrapolate from the trajectory of a baseball, we would say that it would continue towards roughly the center of the earth but it doesn't.