I think this is a foot-gun, it's a bad idea even if it works great, and I doubt it works very well. You should manage your build artifacts explicitly, not just jam them in git along with the code that generates them because you are already using it and you haven't thought it through.
I don't think you've made your case here. The practices you describe are partly an artifact of computation, bandwidth, and storage costs. But not the current ones, the ones when git was invented more than 15 years ago. In the short term, we have to conform to the computer's needs. But in the long term, it has to be the other way around.
You're right! It makes way more sense, in the long run, to abuse a tool like git in a way that it isn't designed for and which it can't actually support, then instead of actually using git use a proprietary service that may or may not be around in a week. Here I was thinking short term.