I asked about this when they presented the project at the Linux Plumbers conference. They replied that it's not really intended to be a security boundary, and that you should not let anyone malicious load these programs.
Given this thread model, I think their project is entirely reasonable. Safe Rust will prevent accidental mistakes even if you could technically circumvent it if you really try.
eBPF's limitations are as much about reliability as security. The bounded loop restriction, for instance, prevents eBPF programs from locking up your machine.
You could still imagine terminating these programs after some bounded time or cycle count. It isn't as good as static verification, but it's certainly more flexible.
If you're doing this kind of "optimistic" reliability story, where developers who stay on the happy path are unlikely to cause any real problems, I don't get what the value of something like this is over just doing a normal Rust LKM that isn't locked into a specific set of helpers.
You can extend the kernel functionality without having to develop a whole kernel module? Just because your module has no memory errors does not mean that it is working as intended.
Further, if you want to hook into specific parts of the kernel, you might well end up writing far more boilerplate instead of just intercepting the one call you're actually interested in and adding some metadata or doing some access control.
I personally am all for a kernel that can do more things for more people with less bespoke kernel modules or patches.
if nothing else, rex makes a good central place to evolve a set of helper code for doing ebpf-like stuff in a rust kernel module. wouldn't be too surprised if it eventually becomes closer to an embedded dsl.
As I understand it eBPF has also given up on that due to Spectre. As a result you need root to use it on most distros anyway, and the kernel devs aren't going to expand its use (some systems are stuck on cBPF).
So it's not like eBPF is secure and this isn't. They're both insecure in different ways.
> Should a microkernel implement eBPF and WASM, or, for the same reasons that justify a microkernel should eBPF and most other things be confined or relegated or segregated in userspace; in terms of microkernel goals like separation of concerns and least privilege and then performance?
I have no insight into the Asahi project, but the LKML link goes to an email from James Calligeros containing code written by Hector Martin and Sven Peter. The code may have been written a long time ago.
That's an email from James Calligeros. All this patch says is that the author is Hector Martin (and Sven Peter). The code could have been written a long time ago.
I was about to take offence at the use of “trivial” in this context. But then I noticed your handle, lol. You have the license to say that, thanks for your contributions!
Won't that be an eager runtime though? Breaking Rust's assumption that futures do nothing until polled? Unless you don't submit it to the queue until the poll call, I guess
> IIRC Alice from the tokio team also suggested there hasn't been much interest in pushing through these difficulties more recently, as the current performance is "good enough".
Well, I think there is interest, but mostly for file IO.
For file IO, the situation is pretty simple. We already have to implement that using spawn_blocking, and spawn_blocking has the exact same buffer challenges as io_uring does, so translating file IO to io_uring is not that tricky.
On the other hand, I don't think tokio::net's existing APIs will support io_uring. Or at least they won't support the buffer-based io_uring APIs; there is no reason they can't register for readiness through io_uring.
This covers probably 90% of the usefulness of io_uring for non-niche applications. Its original purpose was doing buffered async file IO without a bunch of caveats that make it effectively useless. The biggest speed up I’ve found with it is ‘stat’ing large sets of files in the VFS cache. It can literally be 50x faster at that, since you can do 1000 files with a single systemcall and the data you need from the kernel is all in memory.
High throughput network usecases that don’t need/want AF_XDP or DPDK can get most of the speedup with ‘sendmmsg/recvmmsg’ and segmentation offload.
For TCP streams syscall overhead isn't a big issue really, you can easily transfer large chunks of data in each write(). If you have TCP segmentation offload available you'll have no serious issues pushing 100gbit/s. Also if you are sending static content don't forget sendfile().
UDP is a whole another kettle of fish, get's very complicated to go above 10gbit/s or so. This is a big part of why QUIC really struggles to scale well for fat pipes [1]. sendmmsg/recvmmsg + UDP GRO/GSO will probably get you to ~30gbit/s but beyond that is a real headache. The issue is that UDP is not stream focused so you're making a ton of little writes and the kernel networking stack as of today does a pretty bad job with these workloads.
FWIW even the fastest QUIC implementations cap out at <10gbit/s today [2].
Had a good fight writing a ~20gbit userspace UDP VPN recently. Ended up having to bypass the kernels networking stack using AF_XDP [3].
I'm available for hire btw, if you've got an interesting networking project feel free to reach out.
Yeah all agreed - the only addendum I’d add is for cases where you can’t use large buffers because you don’t have the data (e.g. realtime data streams or very short request/reply cycles). These end up having the same problems, but are not soluble by TCP or UDP segmentation offloads. This is where reduced syscall overhead (or even better kernel bypass) really shines for networking.
These are per-connection bottlenecks, largely due to implementation choices in the Linux network stack. Even with vanilla Linux networking, vertical scale can get the aggregate bandwidth as high as you want if you don’t need 10G per connection (which YouTube doesn’t), as long as you have enough CPU cores and NIC queues.
Another thing to consider: Google’s load balancers are all bespoke SDN and they almost certainly speak HTTP1/2 between the load balancers and the application servers. So Linux network stack constraints are probably not relevant for the YouTube frontend serving HTTP3 at all.
I'm quite careful to tightly control the dependencies of Tokio. All dependencies are under control by members of the Tokio team or others that I trust.
Regarding the review process ... one thing that I find challenging and don't know a good solution to is documentation. I've received many PRs where the change itself is fine, but the PR is dragging out because the documentation is lacking, and getting the PR author to improve it sometimes takes a lot of review rounds.
What would you do to avoid this?
Sometimes the same situation comes up with tests, but it is not as common in my experience.
Get people to write the docs first. Not many people like writing docs after the fact, and much of the value of working documentation is lost if you do it after the implementation.
Assuming we’re not taking about user guide kind of docs, then a major benefit of writing docs first is to clarify your thinking. Being able to explain your intent in the written word is valuable because you will often uncover gaps in your thinking. This applies to a specification, or to acknowledging problem reports and updating with theories on what the cause of said problem is and an approach to confirming or fixing it. You can even reference that problem report in commits and merge requests. It pretty beneficial all around.
And docs don’t have to me masterpiece works of art. Just getting people to clarify intent is a huge win. Peer reviewers don’t have time to do a super deep dive into code. If they know what you intended code to do, that’s something many reviewers can check pretty quickly without having to know much context.
It’s selfish and naive to disregard basic documentation of intent.
One option would be to take an initial stab at the documentation yourself - that makes it clear to the submitter where things are unclear, because you made mistakes or omitted things, and they can just correct that, which is a lot more feasible to do than figuring out what's important while your head is in the code.
Given this thread model, I think their project is entirely reasonable. Safe Rust will prevent accidental mistakes even if you could technically circumvent it if you really try.