git-xet follows the patterns established by other git extensions like git-annex and git-lfs, but with some UX improvements. In all of these cases, the large/binary file contents are actually stored outside of git, but available with git commands. The goal here is to make working with and versioning code and data together seamless, not to blindly use the git internals to store data. Does that still seem like a bad idea to you?