Complexity. It's now playing catch up trying to support all of these different chips with all of these different peripherals, capabilities, and errata in quick succession.
As a developer, it's much more difficult to navigate when there are 8 files with the same name, 5 registers with the same name defined in different files, etc. A lot of code is heavily punctuated by ifdefs to selectively include and exclude lines for different targets, making it more difficult to follow. Intellisense struggles, no matter how well it's configured.
I could deal with all that but it's the runtime cost that is the worst. ESP-IDF is constantly getting bigger and slower. All of these abstractions, structs half-full of pointers being handed about at runtime, dead code being linked in for features the chip doesn't even support, it all has a cost.
This is even worse in binary blobs where conditional compilation is more difficult to do. I'm not convinced they're even trying though. A single commit that only mentioned WiFi 6 changes increased my binary size by over 10kB. On a chip that doesn't support WiFi 6.