The CRITICAL vulnerability here is the development process. A lot of projects get by due to a really frickin' good project lead, really good contributors, and really good collaboration. Clearly that's not the case with OpenSSL.

I've suggested in the past that the OS should handle transport encryption. People moan "oh no then we can't fix anything! oh no we can't innovate!". But adding encryption routines to the OS does not remove the ability to use OpenSSL. People will continue to invent their own userland tcp/ip stacks regardless. But having the OS team handle the encryption at least gives one large well-funded organization the burden of doing it right or being really embarrassed. The upshot is every application can simply pick up encryption functionality without depending on a 3rd party library. A new flag to existing syscalls could wrap opening any socket with some standard encryption method, so the application would never need to bother with "doing encryption", unless it wanted to. And the system keychain can manage credentials. The whole point of the OS is to make life easier for apps and users; why not let it do more of the heavy lifting?

And then every time there's a vulnerability in the kernel implementation, we need a kernel patch (something that often takes longer to test and release than an update to a userland library) and a reboot. Updating OpenSSL just requires restarting daemons that use it.

At any rate, I don't see why you think that tying TLS to the kernel is required to improve the security posture. OpenSSL can remain a library, and the same companies that fund the Linux kernel can fund OpenSSL. Hell, the people who maintain the existing crypto functionality in the kernel can work on OpenSSL if they so choose.

Then there's the matter of adoption: the main reason that LibreSSL has failed on Linux is because developers don't seem to want to move to it in their applications, despite it having very few differences from OpenSSL. What makes you think developers will adopt a completely different Linux-specific API that doesn't work on any other OS?

OpenSSL is part of the OS as far as users are concerned. It's right there after installation, in /usr/bin/openssl and in /lib64/libcrypto.so.3. Applications normally just pick it up. And indeed it does not prevent applications from shipping it separately.

But, depending on the OS you're using, the organization is not necessarily well-funded. Even those who are and use OpenSSL (Red Hat) haven't caught this one.

I assume that you're not mixing up "OS" and "kernel", because adding more attack surface to a monolithic kernel is never a good idea.

I mean the kernel. But there's a lot of different ways to implement this functionality. The kernel already has a variety of crypto baked in, it's just a matter of deciding on the implementation, which doesn't have to be 100% in-kernel. The point is to offload the maintenance onto a team that is better staffed and has better development practices, and possibly also add additional functionality to existing syscalls.

I think those are two separate problems. When you want to benefit from a process (better staffed project), you don't need to impose technical conditions (make something part of syscalls). While the Linux kernel doesn't publish userspace OS components (I imagine there's overlap with projects like glibc), I believe Windows "kernel" is composed of many pieces, including some running in userspace, and that would be the organizational entry for your idea.

Linux kernel publishes util-linux, iproute2 and perf at the very least. I don’t see why they couldn’t publish an so.

Basically you’ve described how TLS works for most native Windows (SCHANNEL) and MacOS/iOS apps. The OS provides a library that apps can depend on being present and maintained. The same library is used by user space and kernel alike.

We'd need some developer in Debian and/or Fedora to step up and propose a switch to Libressl.

The rest of the ecosystem would simply follow, as maintaining an alternative is extra work, and, as openssl is simply worse, pointless.