> critics love the Twitter example. But look closer. Ruby carried them further than most companies will ever reach. They outgrew their shoes. That’s not an indictment… that’s success.
> I’ve never seen a team fail because they chose Ruby. I have seen them fail because they chose complexity. Because they chose indecision.
> GitHub held the world’s source code together for years using Ruby.
There are many examples of companies that used Ruby at one point very successfully but moved on from it once it no longer fit their situation. This isn't a critique of Ruby! But it is agreeing that Ruby can be outgrown and that, if you are looking to start with a language your usecase probably won't ever outgrow, Ruby might not be the best choice.
Neat. How is offlineasm used? (Without going into the details about the background of LLInt, that is—what I mean is, how is the compiler invoked?) Is it just the reference compiler, corresponding to some other machinery inside JSC?
That’s how the interpreter in JavaScriptCore gets compiled. The interpreter is written in a macro assembly dialect I invented and this is the compiler for it.
(I say compiled, not assembled, because it’s higher level than normal assembly. There’s an actual pipeline of transformations that happens. Plus a Turing complete macro language)
- Not ABI compatible with Yolo-C. So, you have to recompile the whole stack. This is both a major catch, and it is both a bug and a feature. It's a bug for obvious reasons. It's a feature because it means I'll probably be the first to have a totally usable, totally memory safe desktop environment.
- In my testing, it's between 1.2x and 4x slower than Yolo-C. It uses between 2x and 3x more memory. Others have observed higher overheads in certain tests (I've heard of some things being 8x slower). How much this matters depends on your perspective. Imagine running your desktop environment on a 4x slower computer with 3x less memory. You've probably done exactly this and you probably survived the experience. So the catch is: Fil-C is for folks who want the security benefits badly enough.
It's not obvious that either of these catches are fundamental. The Fil-C development philosophy is very much to get something working first, which means downscoping. That's why I didn't include Yolo-C ABI compat as a goal at all and it's also why I haven't done as many optimizations as I should have. If you look at my GitHub you'll see >20 open issues with performance optimization ideas.
> In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
Of course this has happened.
Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C). Fil-C doesn't have an `unsafe` statement and the dependencies are all recompiled with Fil-C.
> Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Have you considered making... Fil-Zig? Would it be easier to do because Zig is already fairly far ahead of C in terms of safety?
> Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C).
To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe", it's just that PAM isn't any safer (because it's unmodified C). Also you can use Rust without libc - https://github.com/sunfishcode/mustang - unfortunately it doesn't seem to have gained much traction which I think is a shame because glibc is a curse.
Most Rust programs actually have very few C dependencies. The big exceptions are libc and OpenSSL, but I think they'll be excised eventually.
> sudo-rs: uses pam compiled with Yolo-C, so it's not actually safe
Well it is, it's just that it doesn't magically make PAM safe either.
We're not disagreeing about anything technical, I just think it's slightly unfair to say Rust isn't as safe as Fil-C based on that. It is as safe; it just can't automatically make all C code safer like Fil-C can (and CHERI, etc.).
Almost all uses of unsafe in Rust are either for FFI, or to avoid the overhead of things like ref-counting. If you use, eg, Rc RefCell everywhere you will achieve safety without issue.
> Almost all uses of unsafe in Rust are either for FFI
Which makes Rust less safe than Fil-C.
Consider that you have a dependency like libc, PAM, openssl, or the like.
In Rust: you will `unsafe` call into the unsafe versions of those libraries. Hence, even if your Rust code is safe in isolation, your program is not safe overall because you're pulling in unsafe code.
In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
> In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
Great work on Fil-C by the way. Very technically impressive stuff!
It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this). More and more C libraries are getting pure rust ports these days. It’s increasingly rare to actually need to pull in C libraries at all. Sticking to safe rust is a bit inconvenient - but I suspect so is using fil-c. I suppose with fil-C you can use any libraries you want, you’ve just gotta recompile them. That’s cool!
Rust has a bunch of unsafe code in std, so you’re still at risk of a bug there causing safety problems. But the same is true of Fil-C.
I could definitely see myself reaching for Fil-C when running legacy code. It’d be nice if there was a way to use fil-C to protect my rust program from memory bugs in C dependencies. But I can think of dozens of ways why that would be tricky to pull off.
Also I’m curious - can Fil-C protect against threading bugs like rust can? Rust has Send & Sync traits to mark which objects can be safely shared between threads. I can’t think of a way to do that in an automated way without help from the programmer.
> It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
Say you wanted to write a tool like sudo in C# or Go. Here's what would happen: you'd end up having to rely on dependent libraries, like pam, which then relies on other things like libselinux and libaudit. And libselinux depends on pcre2. Those things would be compiled with Yolo-C, so while your sudo tool's new code written by you would be safe, the process would mostly contain unsafe code.
But if you write sudo in Fil-C, then you can rely on all of those dependencies being compiled with Fil-C. The whole process is safe.
So, the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
> If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this).
That's impractical for the reasons above.
> Also I’m curious - can Fil-C protect against threading bugs like rust can?
Not like Rust can, but yes, it does.
Fil-C preserves memory safety under races. This means that debugging multithreaded C/C++ programs in Fil-C is a much nicer experience than in Yolo-C. If you do something wrong, you're going to get a panic - similar to how multithreading bugs in Java or C# lead to exceptions.
But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
> the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
Right; that makes a lot of sense. So currently the only implementation of pam is written in Yolo-C. If you want to build a safe, secure tool which depends on pam - like sudo - then you either need to trust pam (like sudo-rs does). Or you need some way to compile pam to safe code. Fil-C is a way to do that.
Just to name them - I can think of two other solutions to this problem:
1. Sandbox pam within a wasm container. This is probably less convenient than Fil-C (although it would allow that wasm container to be linked from yolo-c, rust, go, js, etc). For pam in particular, this would also be much less safe than Fil-C, since memory bugs in pam would still be possible, and might still result in privilege escalation. Wasm only sandboxes memory at the boundary. It wouldn't protect pam itself from its own bugs like Fil-C does.
2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
I think this also reaffirms the role I see for Fil-C: as a tool to make legacy C code (like pam) safe, at the cost of performance. For new code, I'd rather pick a fully GC language if I don't care about performance, or use rust if performance matters. Again, it would be nice if I could have the best of both worlds - use fil-c to wrap and sandbox pam, but write the rest of my program in rust or C# or Go or something else that is already safe without needing to pay fil-c's cost.
> But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
It depends what you're doing, but this can be a pretty big deal. I suppose this puts Fil-C in about the same safety and performance camp as Go, Java and C#. Ie, it has some protections that rust is missing - like there aren't any unsafe blocks. And its missing some protections rust has - like better thread safety. The unique selling point is it works with existing C code, like old C libraries that we don't want to rewrite. Thanks for your work on it!
> 2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
OK, let's think about that for a moment.
Go and Rust do not have dynamic linking. Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
C# only has dynamic loading if you aren't using the AOT. That means you're JITing. I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
And pam is engineered around dynamic linking. That's sort of the whole point of how it works. Your pam stack refers to modules that get dynamically loaded, which then allows pam modules to pull in things loads of other libraries.
So: there is currently no scalable alternative to Fil-C for making pam safe.
.NET Native is essentially dead. NGEN is specific to .NET Framework. It is possible to dynamically link multiple binaries with the same runtime with NativeAOT but it is very exotic and undocumented path that is not guaranteed to be stable. Not much worse than Go's dynamic linking which also requires the same compiler version.
In theory, in practice the almost resourless WinUI/WinAppSDK team is yet to change the minds of the few companies that believed in UWP to move away from it, due to the amount of missing features after 5 years of Project Reunion.
In fact, the ones that care to port their applications might create Web versions than bother with Native AOT.
> Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
Rust has full blown safe dynamic linking - so long as you use rust, and the the same compiler version, for everything. What it doesn't have is runtime libdl support for that. Which would be a problem for the current pam architecture.
I'm not particularly convinced you couldn't just re-architect pam to use static linking though. Or a scripting language.
> Rust has full blown safe dynamic linking - so long as you use rust, and the the same compiler version, for everything.
This is UB in rust as far as I understand:
1) you compile bar using version 1.5 of libfoo.so at link/compile time
2) you update libfoo.so to version 1.6. It has a lot of changes, but is still source compatible and some kind of reasonable precautions were taken, like that struct layouts are the same
3) you run bar with the new libfoo.so version 1.6
This is UB even if you use the same compiler.
If this is UB then that’s not “full blown safe dynamic linking”.
Note that Fil-C does safely support this. This is safe in Fil-C even if you don’t take appropriate precautions (like if you change function signatures, change struct layouts, etc - it’ll still be safe and have well defined outcomes in Fil-C).
What you're looking for is a stable ABI, not just dynamic linking. Honestly not sure if rust guarantees a stable ABI under some set of "reasonable" precautions... likely not but it's never been something I've needed when I've even wanted in the context of matching rust's other constraint of matching compiler versions.
Does Fil-C support dynamic linking to "yolo C" code or only modules that have also been compiled using fil-C? Would all the pam modules (eg pam_systemd.so, pam_selinux.so, etc) also need to be compiled with fil-c? I assume so, to make it safe?
In general dynamic linking support is something that rust, go and C# could definitely improve upon. As I understand it, Rust does support dynamic linking for pure rust code. It just doesn't have a stable ABI yet. So dynamic linking is only supported within the same version of the rust compiler.
Taking a look at pam, it looks like there's 2 parts:
- The configuration parser & dynamic loader, which would be trivial to rewrite in rust or C# or something.
- The actual dynamically loaded modules, and the ABI bridge to call into them.
I think the obvious way to port pam to rust or C# would be to rewrite the core. Then just statically link in all the standard, built-in pam modules (like pam_unix). And - at least for now - handle "external" modules in exactly the same "yolo C" dyld way they are handled today. Yes, that means a dozen lines of boilerplate unsafe rust code across the C ABI bridge. It wouldn't be quite as safe as Fil-C or C# JITting for everything. But its within my personal safety paranoia tolerance. It would certainly be much safer than pam is today, given currently every single line is in yolo mode. It would also mean those modules wouldn't need to all be rewritten in rust all at once, or at all.
If you want a fully safe replacement for the pam bridge maybe wasm+wasi could work here? I'm not sure. I certainly hear you that this is a problem in rust, go and c#, and I'm not entirely sure what a good solution would look like. Well, other than rust stabilizing its ABI. At the pace rust moves at, that could be any decade now.
> I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
Forget every program for a moment. C#'s JIT would certainly be fast enough for pam. And most linux programs don't make such heavy use of dynamic linking to work. So they could be AOT compiled.
On the topic of performance, I'm not sure I want to pay the fil-C performance overhead for all 500+ processes on my system either. I assume the performance ceiling for fil-c is going to be somewhere around the performance of other well optimized GC runtimes. I don't know of any GC language that does better than about a 2x memory usage overhead.
Call me crazy, but for general computing, I like the speed of native code.
> As I understand it, Rust does support dynamic linking for pure rust code. It just doesn't have a stable ABI yet.
Not quite. To safely run a Rust program against a Rust dynamic library, you had to have compiled and linked that program against exactly that version of the Rust library. This is necessary for fundamental Rust type system reasons. It’s not simply a matter of stabilizing ABI. It’s not like updating the library without relinking would work if you used exactly the same rustc version - it would still be unsafe.
So no, Rust doesn’t support dynamic linking in any meaningful way.
> And most linux programs don't make such heavy use of dynamic linking to work.
This is not true at all.
Linux relies on dynamic linking extensively in two ways:
- A dynamic library may be updated independently of its dependents. You can’t do this in Rust, see above.
- Lots of Linux programs rely on the dynamically loaded module architecture like what pam does. All of the scripting language runtimes do this for example.
> Linux relies on dynamic linking extensively in two ways:
>
> - A dynamic library may be updated independently of its dependents. You can’t do this in Rust, see above.
This isn't a linux thing. Its a popular linux distribution thing. Linux distributions want to be in charge of which version of which dependencies every package uses. But that means the same piece of software on different linux machines will use subtly different versions of all their dependencies.
Distribution maintainers love this, because they can patch libjpeg to fix a bug and it'll affect all programs on the system. But its hell for the actual developers of those packages. If I use a dependency at version 1.2, ubuntu might silently replace that with version 1.1 or 1.2.1 or something else - none of which I've actually tested. Debian might only have version 0.9 available, so they quietly try and patch my software to use that instead. If there's a bug as a result, people will blame me for the bug. Sometimes the debian maintainers maintain their own patch sets to fix bugs in old versions - so they ship version 0.9.0-1 or something. When a user is having problems its now almost completely impossible for me to reproduce their environment on my computer.
Linux doesn't have to do any of this. It all works great with static linked binaries. Eg alpine linux uses musl instead of glibc and just statically links everything. This provides better performance in many cases because library code can be inlined.
Apt is in theory language agnostic. But in practice, this whole dynamic linking thing only really works for pure C libraries. Even C++ is a bit too fragile in most cases for this to work. (I think C++ is in theory ABI compatible, but in practice it only seems to work reliably if you compile all binaries with the same compiler, against the same version of the C++ STL.)
The reason Go doesn't support dynamic linking is that Rob Pike thinks its stupid. That dynamic linking only ever made sense when we didn't have much RAM, and those days are way behind us. I think I generally agree with him. I have no idea where that leaves rust.
> This isn't a linux thing. Its a popular linux distribution thing
You mean: it’s an OS thing.
Windows relies on it. MacOS and iOS rely on it.
When shared frameworks get to the multiple GB size then you can’t boot the system without dynamic linking.
> Even C++ is a bit too fragile in most cases for this to work. (I think C++ is in theory ABI compatible, but in practice it only seems to work reliably if you compile all binaries with the same compiler, against the same version of the C++ STL.)
Maybe about a quarter of the shared libraries, and maybe half of the large ones (as measured by source size) on Linux are C++.
It’s true that the preferred idiom is to use C ABI around the edges, but not everyone does this.
C++ has stable ABI even if you use different compilers these days. That’s been true for at least a decade now.
> The reason Go doesn't support dynamic linking is that Rob Pike thinks its stupid
I don’t care about what Rob Pike thinks is stupid.
> That dynamic linking only ever made sense when we didn't have much RAM, and those days are way behind us
Are you ok with me making thay same argument in favor of Fil-C’s current weak sauce optimization story?
“Optimizing your language implementation only even made sense when we didn’t have much RAM and when computers were slow, and those days are way behind us.”
What shared frameworks are multiple gigabytes in size?? If that’s the case I don’t think the problem is dynamic linking. A hoarder shouldn’t solve their problem by renting more houses.
Honestly I’ve been thinking of making my own toy OS and I’m still a bit on the fence about dynamic linking. As I understand it, macOS puts a lot of their UI components into dynamic libraries. That lets them update the look and feel of the OS between versions by swapping out the implementation of the controls for all apps. I think that’s a good design. I like that they have some way to do that. And for what it’s worth, I agree with your earlier point about scripting languages and whatnot using dynamic linking in Linux for extensibility.
In general I want more programs to be able to do what Pam does and be programmatically extensible. Dynamic linking is a very efficient way to do that. I think it’s really cool that Fil-C can transparently provide safety across those API boundaries. That’s something rust can’t really do until it has a stable ABI. In practice it doesn’t usually take much code to wrap / expose a C api from rust. But it’s inconvenient and definitely unsafe.
> Are you ok with me making thay same argument in favor of Fil-C’s current weak sauce optimization story?
No of course not. Static linking is a mixed bag optimisation wise because - while code is duplicated in ram, doing so also allows inlining and dead code elimination. So it’s usually not that bad in practice. And binaries are usually tiny. It’s an application’s data where I want efficiency.
I think Fil-C’s optimisation story is fine for many use cases. It should be able to be as efficient as C#, Go and other similar languages - which are pretty fast languages. More than fast enough for most applications. But there’s a lot of software I’d like to keep in fast native code, like databases and browsers. So I wouldn’t use Fil-C there. And in the places I don’t mind slower code, I’d rather use a C# or go or typescript. So for me that leaves Fil-C as an interesting tool primarily for sandboxing old C code.
If that was how Fil-C was talked about and positioned, that would be fine. But that’s not the narrative. I’m frustrated with some of the hypocrisy I’ve seen online from “influencers” like Casey M. Last week C and systems code was good because of how fast bare metal C runs. But now C is still great because it’s memory safe through Fil-C, so why even bother with rust? What liars! They just like C and don’t want to admit it. If they really cared about performance and systems code, they wouldn’t find Fil-C exciting. And if they actually cared about safety they wouldn’t have been so dismissive of memory safe languages before Fil-C came out. Liking C is fine. Hating rust is fine. There’s a lot to dislike. I just don’t like being lied to.
“I like Fil-C because it lets the Linux ecosystem have safety without needing to change to another language. I like C and want us to use it forever” is - I think - what a lot of people actually believe. If that’s the case, I wish people would come out and say it.
People use C not because they like it, but because its the lingua franca [1]. As long as The Unix Programming Environment [2] remains dominant, The C Programming Language [3] will too.
The ability to replace libraries without recompiling downstream dependents is not a necessary component of dynamic linking. However if rust wanted to gain this ability, it basically is just a matter of a stable ABI. The only other thing you would need is to get a hash of all the parts of the system that are equivalent to C header files and have the program check the hash in the dylib matched the version it was compiled against at load time.
I believe in fact the sdylib crate type in nightly rust does (or at least is intended to do, not sure it's fully implemented) exactly this.
It's not a bug, it's just a lack of an additional feature on top of dynamic linking.
In fact it prevents bugs as it provides a clear bar on the misuse of dynamic linking as is so common in C/C++/(as designed) Fil-C to replace libraries with likely-ABI-incompatible replacements. Fil-C limits the blast radius of bugs caused by this, but doesn't prevent the bugs. Just saying "you can't do that" like current rust actually prevents them. (Alternatively growing that feature properly, including checking ABI compatibility, as intended with sdylib's would also prevent them. To the best of my knowledge no C-like language does this today)
Dynmaic linking used responsibly to reduce linking times in development cycles, and to reduce memory usage when you have memory things using the same library, remains perfectly safe without this feature. The former is reasonably common in the rust ecosystem today, and I've literally never hit or heard of anyone else hitting a safety issue as a result. Minimizing memory usage is a niche use case in the modern world... but it's there if you want it. You just have to not modify the application post compilation - just like you have to not modify any application post compilation to retain memory safety guarantees.
I mean, that's a real advantage. But personally I'm not so keen for all my system software to have the performance of garbage collected languages. I don't want my software to suddenly get 1.5x-4x slower and start using twice as much RAM. For pam and sudo it wouldn't matter, sure. But across the whole linux system? I'd rather a slow and careful rewrite in rust over that sort of system wide slowdown.
There will never be a slow and careful rewrite in Rust. Nobody wants to do it, and doing it without introducing new bugs and security issues (not linked to memory) is even more difficult.
> There will never be a slow and careful rewrite in Rust. Nobody wants to do it
Lots of people want to do it. Whether you want to run any of their code is an open question - but rewriting old things from scratch is a very popular passtime.
In fact, people are already doing it. For example, the recent sudo-rs rewrite. Or the uutils coreutils package:
> doing it without introducing new bugs and security issues (not linked to memory) is even more difficult.
Difficult, but not impossible. Plenty of alternate C compilers exist. And alternate SQL databases. And some web browsers. And email servers. And TLS implementations. And on and on. Arguably there already are several reimplementations of the "linux userland", with various tweaks. They're just called Freebsd. And Illumos. And MacOS.
Why would the linux userland be uniquely impossible to reimplement in another language? I feel like people celebrate when someone makes a new text editor or gameboy emulator. Why so much hate at the idea of more implementations of unix tools?
Except ChromeOS and webOS are both based on linux, which is written in normal, fast C.
The computer on my desk cost about $3000. If all my programs ran twice as slow, thats sort of like removing $1500 of my computer away. Could I get away with a $1500 computer? Probably. But I'm not going to be happy about it.
If you are happy to pay the cost of a garbage collector, why C at all and not Java, C#, Go, Typescript, or any of the other fine options? They’re simpler languages with better tooling and cleaner syntax. I understand Fil-C for running legacy code. But for new code, the only reason I’d ever pick C is performance and Fil-C takes that benefit away.
For legacy code that no one is going to rewrite, of course.
For new code, I would go further than that, first of all languages can have a GC and have C like performance, it is a matter of implementation, algorithms, data structures and design application with mechanical affinity in mind.
I would go for what the last OS designed by UNIX and C authors, C only gets to be used in the kernel, and some more low level drivers, everything else is managed, like Inferno and Limbo.
Many fail to realise that UNIX and C authors moved on from their original creations, improving what they saw as flaws.
> More and more C libraries are getting pure rust ports these days.
I don't think these rewrites will ever be mainstream. There is so much C out there, and typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
> I don't think these rewrites will ever be mainstream.
Forever is a long time. And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
Whether or not new implementations become valuable and get maintained over time is an open question. But the energy is definitely there. (Even though its not necessarily pointed in the direction of economically useful, long lived software.)
> typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
I hear a claim like: "The current C version of most tools is the only place, and only project which will ever bother solving its chosen problem properly." I'm not sure if tahts quite what you mean, but its a wildly bold claim. Even in C and C++, this seems already false. There's several good, solid, standards compliant SQL databases. And OS kernels. And compilers. And web browsers. If there's room for several C based OS kernels, do you really think there'll never be a viable OS kernel in rust, Zig or Odin? Or a web browser or compiler?
> And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
The tech landscape was completely different back then. Linus himself stated that had BSD been free at the time, he wouldn't have started working on a free MINIX clone. He worked to solve a concrete problem (the lack of a free UNIX-like kernel), he didn't partially rewrite something that exists and works well just to "create his own ecosystem."
Ok, that might be true of Linus himself. But I'm not sure what point you're making by bringing that up. Give up then? Linux forever, lets never try and iterate and improve?
Its weird talking to people in this thread. I get the impression lots of folks would really like it if the actual C code making up linux stayed in place forever. Like it would be a good thing if humans exploring the cosmos in the year 3025 were still logging in using pam, written in C. And not just C, but the current lines of code we have right now. Why? Because! Because C is the best at things. Things like dynamic linking! Because don't change what works. Because if you rewrite something you might get bugs. Because C is fast and efficient. Oh, never mind fast and efficient - lets compile it with Fil-C even though it makes it really slow, just so long as we don't have to change the code!
I think gnu/linux could be way better than it is now, in lots of ways. I get the impression that we - as a species - have barely scratched the surface of the design space for operating systems. The UNIX model is a really early design attempt. Is it the best design that will ever exist? Of course not. Do you know how the unix socket API came about? It was an intern / grad student at berkley who happened to be at the right place and right time. They invented an API and its basically never been changed since then. Because its the right API? No. Just because they were there and it was too hard to change it later.
I want proper capabilities. I want a microkernel. I want faster syscalls. I want storage that's relational, not just file based. I want atomic disk operations. I want programs which can be migrated between computers while they're running. I want supervisor trees. I want data that can be moved or shared between computers. I want my programming languages which can talk to each other using something richer than the C ABI. I want debugging tools that work with lots of languages. I want to be able to download and run programs from the internet safely, without risking all my data. This is all technically possible today.
But the idea of just making a bug for bug compatible, drop in replacement of a single tool like sudo is met with so much hostility and distrust from the community. Sometimes I think I live in a different world than most people. I think things can be better. I want things to be better. But we can't get there if we can't accept any change.
> On the other hand fil-c won't help with dependencies written in C++, Fortran, and lots of others.
IIRC Fil-C is already documented to work with C++. It's not immediately obvious to me that there are any fundamental reasons Fil-C wouldn't work for Fortran and "lots of others" as well; a fairly large chunk of its work is done via a pass on LLVM IR [0] so anything that compiles to LLVM could reap Fil-C's benefits with (probably?) relatively minor alterations to frontends.
Pretty sure a vfprintf-like function sits at the bottom of the printf stack in all of the libc's I've surveyed (which includes BSDs). And yeah, BSDs also support memstream APIs, for example https://man.openbsd.org/fmemopen.3
Here's a realistic path for how AI "replaces"/"displaces" a large chunk of the workforce:
- Even without AI most corpos could shed probably 10% of their workforce - or maybe more - and still be about as productive as they are now. Bunch of reasons why that's true, but here are two I can easily think of: (1) after the layoffs work shifts to the people who remain, who then work harder; (2) underperformers are often not let go for a long time or ever because their managers don't want to do the legwork (and the layoffs are a good opportunity to force that to happen).
- It's hard for leadership to initiate layoffs, because doing so seems like it'll make the company look weak to investors, customers, etc. So if you really want to cut costs by shedding 10%+ of your workforce and making the remaining 90% work harder, then you have to have a good story to tell for why you are doing it.
- AI makes for a good story. It's a way to achieve what you would have wanted to achieve anyway, while making it seem like you're cutting edge.
Reason 3: those people are mostly buffer to absorb variable workloads. Firing them increase efficiency at the expense of being unable to keep up with spikes in demand. Productivity will stay about the same until the next crisis hits, then drop.
Running a shop at maximum productivity is not sustainable of course. Quality and morale suffers, best workers leave, and turns out you really need slack for things to work well. ("You should overprovision your capacity" for the engineer mindset)
I wonder if AI also reveals unnecessary parts of the workforce by demonstrating that what they do is actually pretty trivial.
There are a ton of basically BS office jobs that could probably be replaced by AI, or in some cases just revealed as superfluous.
We need to just stop pretending we still need a 1:1 connection between employment and income and do UBI. Useless jobs help us preserve the illusions of a pre-post-industrial civilization. Instead of just paying people, we pay people to do work we don't need.
The joke about someone using chatGPT to write a lengthy email that the recipient will summarize with ChatGPT is the perfect example of how pretend much work is.
Something went wrong once. Maybe not even in your organization, but it went wrong somewhere. Someone added a process to make sure that the problem didn't happen again, because that's what well-run organizations are supposed to do.
But too often, people don't think about the cost of the procedure. People are going to have to follow this procedure every time the situation happens for the next N years. How much does that cost in peoples' time? In money? How much did the mistake cost? How often did it happen? So was the procedure a net gain or a net loss? People don't ask that, but instead the procedure gets written and becomes "industry best practice".
(And for some industries, it is! Aviation, medical, defense... some of those have really tight regulation, and they require strict procedures. But not every organization is in those worlds...)
So now you have poor corporate drones that have to run through that maze of procedures, over and over. Well, if GPT can run the maze for you, that's really tempting. It can cut your boredom and tedium, cut out a ton of meaningless work, and make you far faster.
But on the other hand, if you are the person who wrote the procedure, you think that it matters that it be done correctly. The form has to be filled out accurately, not with random gibberish, not even with correct-sounding-but-not-actually-accurate data. So you cannot allow GPT to do the procedures.
The procedure-writers and procedure-doers live in different worlds and have different goals, and GPT doesn't fix that at all.
There is this joke about socialism where hundreds of workers digging with shovels and somebody asks “Why not use that excavator? One machine could do it in no time” and the other answers “And put 20 men out of work? We’re creating jobs!”.
This is why a lot of modern leftists are anti-tech. Tech destroys jobs. If we are going to maintain the fiction that full employment is necessary for a modern civilization, everyone has to have a job, and for that to be true we have to restrict our technological progress.
Which is really just making a ton of people waste their time doing bullshit work. I fail to see how this is progressive.
Author of the original WTF::ParkingLot here (what rust’s parking_lot is based on).
I’m surprised that this only compared to std on one platform (Linux).
The main benefit of parking lot is that it makes locks very small, which then encourages the use of fine grained locking. For example, in JavaScriptCore (ParkingLot’s first customer), we stuff a 2-bit lock into every object header - so if there is ever a need to do some locking for internal VM reasons on any object we can do that without increasing the size of the object
> The main benefit of parking lot is that it makes locks very small, which then encourages the use of fine grained locking. For example, in JavaScriptCore (ParkingLot’s first customer), we stuff a 2-bit lock into every object header - so if there is ever a need to do some locking for internal VM reasons on any object we can do that without increasing the size of the object
IMHO that's a very cool feature which is essentially wasted when using it as a `Mutex<InnerBlah>` because the mutex's size will get rounded up to the alignment of `InnerBlah`. And even when not doing that, afaict `parking_lot` doesn't expose a way to use the remaining six bits in `parking_lot::RawMutex`. I think the new std mutexes made the right choice to use a different design.
> I’m surprised that this only compared to std on one platform (Linux).
Can't speak for the author, but I suspect a lot of people really only care about performance under Linux. I write software that I often develop from a Mac but almost entirely deploy on Linux. (But speaking of Macs: std::mutex doesn't yet use futexes on macOS. Might happen soon. https://github.com/rust-lang/rust/pull/122408)
Hypothetically Rust could make `Mutex<InnerBlah>` work with just two bits in the same way it makes `Option<&T>` the same size as `&T`. Annotate `InnerBlah` with the information about which bits are available and let `Mutex` use them.
There was talk of Rust allowing stride != alignment. [1] I think this would mean if say `InnerBlah` has size 15 and alignment 8, `parking_lot::Mutex<InnerBlah>` can be size 16 rather than the current 24. Same would be true for an `OuterBlah` the mutex is one field of. But I don't think it'll happen.
In principle, you Rust could create something like std::num::NonZero and its corresponding sealed trait ZeroablePrimitive to mark that two bits are unused. But that doesn't exist yet as far as I know.
There are also currently the unstable rustc_layout_scalar_valid_range_start and rustc_layout_scalar_valid_range_end attributes (which are used in the definition of NonNull, etc.) which could be used for some bit patterns.
I do the same in my toy JVM (to implement the reentrant mutex+condition variable that every Java object has), except I've got a rare deadlock somewhere because, as it turns out, writing complicated low level concurrency primitives is kinda hard :p
How can a parking_lot lock be less than 1 byte? does this uses unsafe?
Rust in general doesn't support bit-level objects unless you cast things to [u8] and do some shifts and masking manually (that is, like C), which of course is wildly unsafe for data structures with safety invariants
I don’t know the details of the Rust port but I don’t imagine the part that involves the two bits to require unsafe, other than in the ways that any locking algorithm dances with unsafety in Rust (ownership relies on locking algorithms being correct)
This is very similar to how Java's object monitors are implemented. In OpenJDK, the markWord uses two bits to describe the state of an Object's monitor (see markWord.hpp:55). On contention, the monitor is said to become inflated, which basically means revving up a heavier lock and knowing how to find it.
I'm a bit disappointed though, I assumed that you had a way of only using 2 bits of an object's memory somehow, but it seems like the lock takes a full byte?
It’s just that if you use the WTF::Lock class the. You get a full byte simply because the smallest possible size of a class instance in C++ is one byte.
But there’s a template mixing
thing you can use to get it to be two bits (you tell the mixin which byte to steal the two bits from and which two bits).
I suspend the same situation holds in the Rust port.
I am very familiar with how Java does locks. This is different. Look at the ParkingLot/parking_lot API. It lets you do much more than just locks, and there’s no direct equivalent of what Java VMs call the inflated or fat lock. The closest thing is the on demand created queue keyed by address.
>I am very familiar with how Java does locks. This is different. Look at the ParkingLot/parking_lot API. It lets you do much more than just locks, and there’s no direct equivalent of what Java VMs call the inflated or fat lock. The closest thing is the on demand created queue keyed by address.
Are you familiar with the new LightweightSynchronizer approach with an indirection via a table, instead of overwriting the markWord? I'd say that this has pushed the ParkingLot approach and Java's (Hotspot's, really) approach closer to each other than before. I think the table approach in Java could be encoded trivially into ParkingLot API, and the opposite maybe. Obviously the latter would be a lot more hamfisted.
The idea is that six bits in the byte are free to use as you wish. Of course you'll need to implement operations on those six bits as CAS loops (which nonetheless allow for any arbitrary RMW operation) to avoid interfering with the mutex state.
Unhelpful response. This cuongle.dev article does not answer nextaccountic's question, and neither do the webkit.org articles that describe the parking lot concept but not this Rust implementation. The correct answer appears to be that it's impossible: `parking_lot::RawMutex` has private storage that owns the entire byte and does not provide any accessor for the unused six bits.
(unless there's somewhere else in the crate that provides an accessor for this but that'd be a weird interface)
(or you just use transmute to "know" that it's one byte and which bits within the byte it actually cares about, but really don't do that)
(slightly more realistically, you could probably use the `parking_lot_core::park` portion of the implementation and build your own equivalent of `parking_lot::RawMutex` on top of it)
(or you send the `parking_lot` folks a PR to extend `parking_lot::RawMutex` with interface you want; it is open source after all)
> and neither do the webkit.org articles that describe the parking lot concept but not this Rust implementation
The WebKit post explicitly talks about how you just need two bits to describe the lock state.
> The correct answer appears to be that it's impossible: `parking_lot::RawMutex` has private storage that owns the entire byte and does not provide any accessor for the unused six bits.
Not impossible. One way to do this is to just use parking_lot directly.
In WebKit there’s a template mixin that lets you steal two bits for locking however you like. JavaScriptCore uses this to steal two bits from the indexing type byte (if I remember right)
> The WebKit post explicitly talks about how you just need two bits to describe the lock state.
It describes the algorithm but not how a caller of the Rust `parking_lot` crate could take advantage of this.
> Not impossible. One way to do this is to just use parking_lot directly.
By "just use parking_lot directly", I think you're talking about reimplementing the parking lot algorithm or using the C++ `WTF::ParkingLot` implementation? But not actually using the existing Rust crate called `parking_lot` described in the cuongle.dev article? That's confusingly put, and nextaccountic's question is certainly Rust-specific and likely expecting an answer relating to this particular crate. At the least, "does this use unsafe" would certainly be true with an implementation from scratch or when using FFI into C++.
I hear that this algorithm and the C++ implementation are your invention, and all due respect for that. I'm also hearing that you are not familiar with this Rust implementation. It does not offer the main benefit you're describing. `parking_lot::RawMutex` is a one-byte type; that six bits within it are unused is true but something callers can not take advantage of. Worse, `parking_lot::Mutex<InnerFoo>` in practice is often a full word larger than `InnerFoo` due to alignment padding. As such, there's little benefit over a simpler futex-based approach.
> It describes the algorithm but not how a caller of the Rust `parking_lot` crate could take advantage of this.
Read the WebKit post.
> By "just use parking_lot directly", I think you're talking about reimplementing the parking lot algorithm or using the C++ `WTF::ParkingLot` implementation? But not actually using the existing Rust crate called `parking_lot` described in the cuongle.dev article?
"just use parking_lot directly" is a weird way to say "use the `parking_lot_core` crate instead of the `parking_lot` crate".
...and note that I mentioned this in my earlier comment: (slightly more realistically, you could probably use the `parking_lot_core::park` portion of the implementation and build your own equivalent of `parking_lot::RawMutex` on top of it)
I'm not trying to be disagreeable here, but really you could save a lot of trouble if you were a bit more careful in your communication.
No. nextaccountic's comment and the cuongle.dev article are both talking about Rust. The Rust `parking_lot` implementation only uses two bits within a byte, but it doesn't provide a way for anything else to use the remaining six.
pizlonator's comments mention both the (C++) WTF::ParkingLot and the Rust `parking_lot`, and they don't answer nextaccountic's question about the latter.
> nextaccountic is confused.
nextaccountic asked how this idea could be applied to this Rust implementation. That's a perfectly reasonable question. pizlonator didn't know the anwer. That's perfectly reasonable too. Conscat suggested the article would be helpful; that was wrong.
Before Perl, there was no scripting language that could do systems tasks except maybe shell and tcl, but that's shell is an extremely unpleasant programming experience and the performance is horrid, and tcl's string-based nature is just too weird.
Perl gives you something more like a real programming language and can do shell-like tasks and systems tasks very nicely. Compared to what came before, it is amazing.
But then Ruby and Python came along and checked the "real programming language" box even more firmly than Perl while retaining the shell/systems angle. Ruby and Python were better than Perl along exactly the same axis as the one on which Perl was better than Tcl and shell.
IMHO Python killed both Perl and Ruby. While Ruby is more alive than Perl it's nowhere near as popular as Python.
I like Perl and used it professionally for year and vaguely remember probably around 2010x relatively massive Python evangelism (lots of articles, conferences, lots of messages from Python adepts on forums e.t.c). One of talking points (no longer needed nowadays) was that Python is backed (sponsored) by Google so Python will be successful and you should not worry about it's future and also if you will choose Python you will be successful (as Google is).
I think Ruby has declined because Rails was its selling point, but Rails was optimized for the world of HTML templates. Once you're writing JavaScript-heavy frontends and mobile apps, Rails isn't giving you much that you can't get from Python or server-side JS.
A few people started using Ruby for command line tools[1] but the community was very focuses around rails. Also Ruby isn't usually part of the standard OS install. So Ruby stayed stuck in it's Rails niche.
For me, this is why python took off. People wanted that lucrative job or receive the reflected glory of a winner, so y'gotta learn python. The rest is just post-hoc justification for why you made that choice passed on as "this language is better because of blah..."
A lot of the justifications don't stack up against serious scrutiny, but are accepted as gospel.
In previous life, worked on large object-oriented Perl. There was a difference between good Perl and the Perl in messy scripts. Good Perl was nice to work in but required discipline to keep organized.
I wonder if there was an earlier point of Perl's demise. Perl 5 came out with flexible object-oriented features, but it took years for packages like Moose to come out and make it nice and usable.
I always thought one of the best and worst things about Perl was the fact that you could build something like Moose with it.
But the bad side was that by the time someone was clever enough to invent Moose, all sorts of other bespoke object systems had been invented and used in the meantime, and your CPAN dependencies used every single one of them.
Good luck getting any two people to agree on a sharp line between programming language and scripting language. Perl seems to swap sides depending on the year people are arguing about it.
Entire enterprises ran/still run on Business BASIC and Delphi code. Billion-dollar fortunes have been made on such code. Those languages are used for serious things all the time.
https://www.epicgames.com/site/en-US/careers/jobs/5723897004
reply