The major source of conflict is that the disto model for c/c++ dependencies is simply not good. Developers only tolerate it because the problem was previously unsolved before linux package managers. We are talking about a time before even autoconf, almost everyone was rolling their own configure scripts, or creating makefiles which self-configured.
It was a mess, and linux package managers were first to actually solve the problem. I fully believe that the explosive growth of linux in the early years had little to do with the kernel, and was much more about the proper package managers introduced by Debian, Red Hat, and friends.
But just because package managers were massively better than nothing, and nothing (universally) better has come along since, doesn't mean they are actually good. The user experience is pretty good. But for developers, the drive to dynamically link every single dependency, and only have one (maybe two) versions of each dependency, and to randomly update dependencies causes massive problems for developers. It only works as well as it does for C/C++ applications because they have gotten used to working within the restrictive framework. And maybe a bit of Stockholm syndrome.
-----------------
Let me tell you a story:
Back in 2014, I was working on Dolphin Emulator (which is entirely c++), trying to fix regressions for a Dolphin 5.0 release. One of these regressions was that on linux, launching a game a second time (or opening the controller config window twice) would cause a crash. I quickly tracked the bug down to a bug introduced in SDL 2.0.3. Most consumers of SDL weren't impacted by this bug, because they only ever initialised SDL once.
The fix was simple enough, it had already been fixed in the latest reversion and so would be fixed whenever 2.0.4 was released. But we wanted to release before that, so how to fix the bug? If this was cargo, I would just point it at a git revision (possibly a custom branch, with just the fix back ported). But that wasn't an option.
I spent ages trying to work out a workaround that avoided the bug, but couldn't find one. I considered including a patched version of SDL with dolphin; Dolphin's source tree already contained a bunch of libraries which we statically linked in. Some patched, others were just there for easier builds on Windows/Mac). But I knew from experience that any distro would simply patch our build system to use the distro version. Dolphin's build system already had an extensive set of manual checks to use the distro version of a library if it existed. Many of these of these checks were contributed upstreams by distro package maintainers.
I considered throwing an error message (twice, both at build and runtime) if dolphin was linked with a bugged version of SDL 2... But I figured the distro maintainers would probably patch out both error message too. Remember, we are talking about a bug that wouldn't even be fixed to the next release of SDL, with an uncertain release date. And there would probably be distros which stuck with the buggy version for way too long (Debian actually avoided the bug, because it was still shipping SDL 2.0.1, before the bug was introduced)
Eventually I got frustrated. And I realised that SDL was only used for controller input, and only on linux/bsd. SDL already wasn't meeting our needs, we had a dedicated core input backend for MacOS and both DInput and XInput backends for Windows because they worked better. I had also just gone though the entire SDL input code, back when I was trying to find a workaround, and had learned how it work.
So... to workaround the limitations of Linux Disto packaging (and a stupid, easy to fix SDL bug), I simply deleted SDL out of the Dolphin codebase. Only took me three days to completely reimplement all the controller input functionality, directly accessing the lower-level linux libraries: libudev for controller detection/hotpugging, open the correct `/dev/input/eventX` file, libevdev for parsing the raw input events.
Three days to implement it. A few weeks of testing to find bugs... Not only did this workaround the bug. But it allowed Dolphin to access controller functionality that SDL wasn't exposing at the time (suddenly, the Dual Shock 4's touchpad would work, and the pressure sensitive buttons on Dual Sock 3). I don't think SDL was even exposing the gyroscope/accelerometers of those controllers at the time.
---------------------
I was very happy with the result, but it was crazy that I was forced to go to such extremes, just because of the linux package manager insistence on enforcing one shared version of a dependency. I'm not sure how common it is for developers to rip-out and replace an entire dependency because of distro packaging, but workarounds for buggy library versions are common.
These days, I'm really enjoying programming with rust; The fact that cargo always gives me the exact version of the package I asked for is a massive win. And why should we give it up, just because distros want to do something worse. And it would be so hard to make the transition, because a lot of rust code has been written under the assumption that cargo will give you the version (or version range) than you ask for.
If you think about it, it's kind of insane that packages will be build and tested against one version of a library, just to have it randomly replaced with a later version, potentially introducing bugs. In some edge cases, packages even get downgraded to an older version between build/test and being installed on a user's system.
---------------------
What's the solution for rust? I have no idea. Distros do have a point that statically linking everything isn't the best idea. Maybe we can reach some kind of middle ground where some dependencies are statically linked and others are dynamically linked, but with explicit support for multiple versions of each dependency co-existing on the system.
Maybe we also add explicit support to package managers to track which dependencies have been statically linked into them, so we can force a rebuild of them all when there is a security issue.
> I considered throwing an error message (twice, both at build and runtime) if dolphin was linked with a bugged version of SDL 2... But I figured the distro maintainers would probably patch out both error message too.
Distro maintainers will not patch in a bug into a program, they will patch out bug from the SDL. :-/
Maybe.... Probably. Especially if I had left a detailed comment explaining why.
But I still would have required an interim solution for everyone building from source until these patched/fixed versions of SDL started shipping in distros. And then how would the warning message tell the difference between the original SDL 2.0.3 and a patched version of SDL 2.0.3?
And we are talking about dozens of distros, multiple chances for something to go wrong.
My point is that I quickly came to the conclusion that ripping out SDL and reimplementing the functionality was the simplest, most conclusive solution.
> But I still would have required an interim solution for everyone building from source until these patched/fixed versions of SDL started shipping in distros.
It was a mess, and linux package managers were first to actually solve the problem. I fully believe that the explosive growth of linux in the early years had little to do with the kernel, and was much more about the proper package managers introduced by Debian, Red Hat, and friends.
But just because package managers were massively better than nothing, and nothing (universally) better has come along since, doesn't mean they are actually good. The user experience is pretty good. But for developers, the drive to dynamically link every single dependency, and only have one (maybe two) versions of each dependency, and to randomly update dependencies causes massive problems for developers. It only works as well as it does for C/C++ applications because they have gotten used to working within the restrictive framework. And maybe a bit of Stockholm syndrome.
-----------------
Let me tell you a story:
Back in 2014, I was working on Dolphin Emulator (which is entirely c++), trying to fix regressions for a Dolphin 5.0 release. One of these regressions was that on linux, launching a game a second time (or opening the controller config window twice) would cause a crash. I quickly tracked the bug down to a bug introduced in SDL 2.0.3. Most consumers of SDL weren't impacted by this bug, because they only ever initialised SDL once.
The fix was simple enough, it had already been fixed in the latest reversion and so would be fixed whenever 2.0.4 was released. But we wanted to release before that, so how to fix the bug? If this was cargo, I would just point it at a git revision (possibly a custom branch, with just the fix back ported). But that wasn't an option.
I spent ages trying to work out a workaround that avoided the bug, but couldn't find one. I considered including a patched version of SDL with dolphin; Dolphin's source tree already contained a bunch of libraries which we statically linked in. Some patched, others were just there for easier builds on Windows/Mac). But I knew from experience that any distro would simply patch our build system to use the distro version. Dolphin's build system already had an extensive set of manual checks to use the distro version of a library if it existed. Many of these of these checks were contributed upstreams by distro package maintainers.
I considered throwing an error message (twice, both at build and runtime) if dolphin was linked with a bugged version of SDL 2... But I figured the distro maintainers would probably patch out both error message too. Remember, we are talking about a bug that wouldn't even be fixed to the next release of SDL, with an uncertain release date. And there would probably be distros which stuck with the buggy version for way too long (Debian actually avoided the bug, because it was still shipping SDL 2.0.1, before the bug was introduced)
Eventually I got frustrated. And I realised that SDL was only used for controller input, and only on linux/bsd. SDL already wasn't meeting our needs, we had a dedicated core input backend for MacOS and both DInput and XInput backends for Windows because they worked better. I had also just gone though the entire SDL input code, back when I was trying to find a workaround, and had learned how it work.
So... to workaround the limitations of Linux Disto packaging (and a stupid, easy to fix SDL bug), I simply deleted SDL out of the Dolphin codebase. Only took me three days to completely reimplement all the controller input functionality, directly accessing the lower-level linux libraries: libudev for controller detection/hotpugging, open the correct `/dev/input/eventX` file, libevdev for parsing the raw input events.
Three days to implement it. A few weeks of testing to find bugs... Not only did this workaround the bug. But it allowed Dolphin to access controller functionality that SDL wasn't exposing at the time (suddenly, the Dual Shock 4's touchpad would work, and the pressure sensitive buttons on Dual Sock 3). I don't think SDL was even exposing the gyroscope/accelerometers of those controllers at the time.
---------------------
I was very happy with the result, but it was crazy that I was forced to go to such extremes, just because of the linux package manager insistence on enforcing one shared version of a dependency. I'm not sure how common it is for developers to rip-out and replace an entire dependency because of distro packaging, but workarounds for buggy library versions are common.
These days, I'm really enjoying programming with rust; The fact that cargo always gives me the exact version of the package I asked for is a massive win. And why should we give it up, just because distros want to do something worse. And it would be so hard to make the transition, because a lot of rust code has been written under the assumption that cargo will give you the version (or version range) than you ask for.
If you think about it, it's kind of insane that packages will be build and tested against one version of a library, just to have it randomly replaced with a later version, potentially introducing bugs. In some edge cases, packages even get downgraded to an older version between build/test and being installed on a user's system.
---------------------
What's the solution for rust? I have no idea. Distros do have a point that statically linking everything isn't the best idea. Maybe we can reach some kind of middle ground where some dependencies are statically linked and others are dynamically linked, but with explicit support for multiple versions of each dependency co-existing on the system.
Maybe we also add explicit support to package managers to track which dependencies have been statically linked into them, so we can force a rebuild of them all when there is a security issue.