Leaving it next to the files means if directories are moved, copied, or renamed, the metadata is still there. So you or I might prefer it to live in a separate file under ~/ but I cannot say littering the whole tree with the metadata buys nothing or can't serve a purpose.
Because macOS doesn't know when files and directories are moved on other devices, network drives, etc.
Nor can it preserve that metadata across other devices, network drives, etc. without copying it across somehow; the easiest solution was to store that data in hidden files and let Finder handle their automatic transfer.
Failure of your imagination isn't failure on someone else's part.
Hmmm this is the first good argument I've heard for the existence of .DS_Store files. You get a point :-)
I'd be curious to test this, by zipping folders, copying them to other machines, etc. My gut feeling is that Apple probably missed something and it doesn't quite preserve the metadata like it should (most likely due to version incompatibilities or duplication of state somewhere else in the OS that overrides it). I'd be pleasantly surprised if it did work though.
It brings up another point however, that probably metadata is as important as data (and sometimes more important, if you look at how we organize data today). The earliest filesystems probably should have implemented multidimensional storage somehow, similar to the metadata in the BeOS filesystem. I'm not familiar with how it's implemented, and it probably also breaks when files are copied to another filesystem, unfortunately:
