For one thing, just admitting more people almost certainly decreases the access to quality education overall (as well as introducing pragmatic difficulties like housing). Schools do tweak who they admit outside of largely artificial measures like SATs. But, within reason, that's not a bad thing. You probably don't want to admit a class of students from top prep schools who had college test prep classes and helped starving children in Africa.
> For one thing, just admitting more people almost certainly decreases the access to quality education overall
Why would that be the case? There are many much larger universities all around the globe and also in the US that manage to provide quality education to their students.
To me, the statements that colleges make about their admission procedures always seem hypocritical to me. The colleges claim that the goal is to advance gender equality and provide education to underrepresented groups (which would not require a small student body) when their main goal seems to be in fact to create a small in-group of people who have made the right connections during their studies (which absolutely does require a small student body).
Schools can be arbitrarily large I guess, e.g. some of the large state schools in the US. But private universities decide on their missions--which are often to have a smaller and more focused student body to your point. Making connections through my studies was mostly never a big deal but having a (somewhat) smaller school was. There are plenty of larger (and cheaper) universities if that's what you're looking for.