Because of how Git's current implementation of DVS works. There's nothing fundamental about it. Git already supports partial clones and on-demand checkouts in some ways, it's just not very ergonomic.
All that's really needed is a way to mark individual files as lazily fetched from a remote only when needed. LFS is a hacky substandard way to emulate that behaviour. It should be built in to Git.
Something like git-lfs is the appropriate solution. You need a little bit of centralization.