Great work and thanks for making it OSS! I was familiar with the prior (academic) work and its limitations, specifically TCP/IP. Could you elaborate on how you solved that problem?
Sure! So it really breaks down into two cases: internal and external networking relative to the container Hermit creates.
(1) internal networking
If you run a test like `rust/network_hello_world.rs` under Hermit, then the communication between threads is part of the "deterministic bubble" that we're running inside of. When one thread blocks on a network call, the Hermit scheduler takes the thread out of the run pool, and it has to deterministically decide when it is ready to rejoin the run-pool by waking up. The scheduler proceeds in linear turns (labeled "COMMIT" in the logs), and if thread 5 unblocks from a network read at turn 100 in one run, it must unblock at that same point in time in all other runs.
Sometimes we use a precise model of the blocking operation (like with futexes) and other times we depend on sending Linux a non-blocking version of the syscall as a way to poll the IO and see if it is ready to complete (given the history of every operation that has committed on turns 1..N-1).
(2) external networking
This is impossible to determinize, of course. Unless you suck the whole network including both hosts into the deterministic bubble, as the DDOS fork of Linux experimented with in ~2013. That was kind of a negative result IMO because performance was pretty bad, but the paper is here:
That's where record-replay comes in. `hermit record` can record network calls, but is in a pretty early state and doesn't support many programs. `hermit run` can just allow networking through and hope for the best, but in the future we plan to add features to record just network calls (and no other syscalls), so that you can mix and match different external-network-responses with different thread schedules. That is, you could "pin" the network responses with network-only recording, and then mess around with other parameters or even modify the program.
So far my attempts to reproduce the alleged performance degradation have not been fruitful. I've written up a fairly detailed gist [1] on how to get CPU performance metrics; appendix also has a dump of C1 and C2 compiled methods (useful for comparison). I also ran on 2-node NUMA; binding cpu and memory to different nodes didn't yield a repro. either.
Inpher (https://www.inpher.io/) | New York (USA), Lausanne (Switzerland), Paris (France) | Software Engineers | Full-time | Onsite
At Inpher, we believe that privacy and security are foundational to the future of computing and have built enterprise products to make this vision a reality.
We are a small team of veteran founders, world-renowned cryptographers and proven software engineers.
We are headquartered in New York City, with satellite offices in San Francisco and Lausanne, Switzerland, and have raised $14M in funding.
Inpher (https://www.inpher.io/) | New York (USA), Lausanne (Switzerland), Paris (France) | Software Engineers | Full-time | Onsite
At Inpher, we believe that privacy and security are foundational to the future of computing and have built enterprise products to make this vision a reality.
We are a small team of veteran founders, world-renowned cryptographers and proven software engineers.
We are headquartered in New York City, with satellite offices in San Francisco and Lausanne, Switzerland, and have raised $14M in funding.
Inpher (https://www.inpher.io/) | New York (USA), Lausanne (Switzerland), Paris (France) | Software Engineers | Full-time | Onsite
At Inpher, we believe that privacy and security are foundational to the future of computing and have built enterprise products to make this vision a reality.
We are a small team of veteran founders, world-renowned cryptographers and proven software engineers.
We are headquartered in New York City, with satellite offices in San Francisco and Lausanne, Switzerland, and have raised $14M in funding.
Inpher (https://www.inpher.io/) | New York (USA), Lausanne (Switzerland), Paris (France) | Software Engineers | Full-time | Onsite
At Inpher, we believe that privacy and security are foundational to the future of computing and have built enterprise products to make this vision a reality.
We are a small team of veteran founders, world-renowned cryptographers and proven software engineers.
We are headquartered in New York City, with satellite offices in San Francisco and Lausanne, Switzerland, and have raised $14M in funding.
Inpher (https://www.inpher.io/) | New York (USA), Lausanne (Switzerland), Paris (France) | Software Engineers | Full-time | Onsite
At Inpher, we believe that privacy and security are foundational to the future of computing and have built enterprise products to make this vision a reality.
We are a small team of veteran founders, world-renowned cryptographers and proven software engineers.
We are headquartered in New York City, with satellite offices in San Francisco and Lausanne, Switzerland, and have raised $14M in funding.
Inpher (https://www.inpher.io/) | New York (USA), Lausanne (Switzerland), Paris (France) | Software Engineers | Full-time | Onsite
At Inpher, we believe that privacy and security are foundational to the future of computing and have built enterprise products to make this vision a reality.
We are a small team of veteran founders, world-renowned cryptographers and proven software engineers.
We are headquartered in New York City, with satellite offices in San Francisco and Lausanne, Switzerland, and have raised $14M in funding.