> I started writing tests in Rust as I would in any other language but found that I was writing tests couldn’t fail.
This is a common refrain in C++ testing: if it compiles then it's probably correct.
> Rust has accounted for so many errors that many common test cases become irrelevant
In practice, if you think this way I think it's a sign that you aren't testing the right things in those other languages. You should be testing business logic, not language stuff.
If you look at your test code and think "I would test this in JavaScript, but I don't need to do that in Rust" then just delete the test.
I’ve definitely experienced it in Haskell and Rust. I can believe some C++ could be that way, but I’ve never experienced it, but then again those projects didn’t have useful tests either. I think with C++ a lot of this depends on domain and the quality of the code and libraries.
A null pointer exception is a bug that breaks business logic. There's no "business logic instead of language stuff" because the language stuff is the foundation that business logic rests on. If you don't test against failure modes, what's even the point in testing?
To close the loop, Rust doesn't include a `null` type and you wouldn't encounter something comparable in idiomatic Rust (because you'd be using eg Option::map to handle None cases gracefully), so this is a class of test that would be common in Java and C that is close to irrelevant in Rust.
To be more specific, Rust does have among other things:
std::ptr::null() - an actual null pointer, probably the zero address on your hardware, and this isn't even an unsafe function. On the other hand, you won't find many uses for a null pointer so, I mean, congrats on obtaining one and good luck with that.
std::ptr::null_mut() - a mutable null pointer, similarly unlikely to be of any use to you in safe Rust, but also not an unsafe thing to ask for.
But, these are pointers, so they're not values that say, a String could take, or a Vec<String> or whatever, only actual raw pointers can be null.
Sure, you can panic unwrapping a None, but there are two important distinctions.
First, this is a controlled panic, not a segmentation fault. The language is ensuring that we don't access the null pointer, or an offset from the null pointer. Null pointer access can be exploitable in certain circumstances (eg, a firmware or kernel). Your use of "exception" suggests you're thinking about it in Java terms however, and Java is equivalent here.
Second, you can only encounter this in explicit circumstances, when you have an Option<T>. Wheras in languages with a null type, any variable can be null implicitly, regardless of it's type.
>First, this is a controlled panic, not a segmentation fault.
But a segmentation fault is also controlled
>Your use of "exception" suggests you're thinking about it in Java terms however, and Java is equivalent here.
No, I am thinking in Delphi terms. It is overspecialized to Windows userspace. Windows gives an Access Violation, and that can be caught, and Delphi throws it as exception
>Second, you can only encounter this in explicit circumstances, when you have an Option<T>. Wheras in languages with a null type, any variable can be null implicitly, regardless of it's type.
Delphi has both nullable types and null-safe types
A segmentation fault may not happen (which is to say, we may corrupt memory or worse) if the null pointer is mapped into memory (as in an embedded or kernel context) or if it's accessed at a large offset, which results in a pointer that's mapped into memory. This may be rotten luck or it may be exploited by an attacker.
> No, I am thinking in Delphi terms.
Fair enough, I don't know Delphi. I'll take what you're saying about it as read.
If an exception breaks business logic, then you can test the code by testing business logic.
How do null pointer exceptions arise?
Inconsistent data: you have code paths that implicitly assume invariants. Trivial cases like “this field is not provided” and more complex ones like “these fields have a specific relationship”.
You can often move those assumptions into the data structure in any language.
Then your tests become a matter of generating and transforming data from a holistic perspective instead of micromanaging individual code paths.
A null pointer exception is a free runtime check. I really don't understand the fuss about null. The "most expansive design mistake in computer science" and whatever.
Free runtime check of what? The point is that if nulls don't exist, there is no need for a "check". Runtime or otherwise. If I write a type that models Foo, I want it to model Foo and not "Foo or null". If I want to model "Foo or no data" then I use a separate type that makes my intention clear (in Rust spelled `Option<Foo>`). Languages where nothing is precisely Foo and everything is "oh by the way this is only possibly Foo" are deeply, deeply flawed.
There is surely something wonderful about the kind of C++ programmer who figures that, since their unusable broken garbage compiled it's probably correct.
Remember unlike most languages you'd be familiar with C++ has IFNDR, which has been jokingly referred to as "False positives for the question: Is this a C++ program?". A conforming C++ compiler is forbidden from telling you in some† unknown number of cases that it suspects what you've written is nonsense, it just has to press on and output... something. Is it a working executable? Could be. Or maybe it's exactly like a working executable except it explodes catastrophically on Fridays. No way to know.
† The ISO standard does identify these cases, but they're so vague that it's hard to pin down everything which is covered. My guess is that all or most non-trivial C++ software is actually IFNDR these days. Just say No to the entire language.
> A conforming C++ compiler is forbidden from telling you in some† unknown number of cases that it suspects what you've written is nonsense, it just has to press on and output... something.
It's not quite that bad - a conforming C++ compiler is permitted to error out and not compile the program. It just doesn't have to.
Many of the cases of IFNDR are semantic constraints, especially in C++ 20 and beyond. As a result of being semantic constraints it's generally impossible to diagnose this with no false positives. The ISO standard forbids such false positives so...
Can you give a more concrete example of the kind of thing you're talking about? Like, if you try to sort something and your comparator implementation for that type is not transitive, the compiler can silently produce a broken binary?
Surely in the undecidable cases the compiler is allowed to produce a binary that errors cleanly at runtime if you did in fact violate the semantic constraint, and any sane implementor would do that. (Not that any sane person would ever write a C++ compiler...)
> Like, if you try to sort something and your comparator implementation for that type is not transitive, the compiler can silently produce a broken binary?
It's not merely about whether your comparisons are transitive, the type must exhibit a total ordering or your sort may do anything, including buffer overflow.
> Surely in the undecidable cases the compiler is allowed to produce a binary that errors cleanly at runtime if you did in fact violate the semantic constraint,
I don't think I know how to prove it, but I'm pretty sure it's going to be Undecidable at runtime too in many of these cases. Rice reduced these problems to Halting, which I'd guess means you end up potentially at runtime trying to decide if some arbitrary piece of code will halt eventually, and yeah, that's not helpful.
I've written about it before, but I should spell it out: The only working alternative is to reject programs when we aren't sure they meet our constraints. This means sometimes we reject a program that actually does meet the constraints but the compiler couldn't see it.
I believe this route is superior because the incentive becomes to make that "Should work but doesn't" set smaller so as to avoid annoying programmers, whereas the C++ incentive is to make the "Compiles but doesn't work" set larger since, hey, it compiles, and I see Rust's Non-Lexical Lifetimes and Polonius as evidence for this on one side, with C++ 20 Concepts and the growing number of IFNDR mentions in the ISO standard on the other side.
> the type must exhibit a total ordering or your sort may do anything, including buffer overflow.
Sure. But there's no requirement for the compiler to be a dick about it, and hopefully most won't.
> I don't think I know how to prove it, but I'm pretty sure it's going to be Undecidable at runtime too in many of these cases. Rice reduced these problems to Halting, which I'd guess means you end up potentially at runtime trying to decide if some arbitrary piece of code will halt eventually, and yeah, that's not helpful.
At runtime can't you just run the code and let it halt or not? Having your program go into an infinite loop because the thing you implemented didn't meet the requirements is not unreasonable.
I'm sympathetic to the idea that there could be a problem in this space, but without a real example of a case where it's hard for a compiler to do something reasonable I'm not convinced.
> I've written about it before, but I should spell it out: The only working alternative is to reject programs when we aren't sure they meet our constraints. This means sometimes we reject a program that actually does meet the constraints but the compiler couldn't see it.
> I believe this route is superior because the incentive becomes to make that "Should work but doesn't" set smaller so as to avoid annoying programmers, whereas the C++ incentive is to make the "Compiles but doesn't work" set larger since, hey, it compiles, and I see Rust's Non-Lexical Lifetimes and Polonius as evidence for this on one side, with C++ 20 Concepts and the growing number of IFNDR mentions in the ISO standard on the other side.
Meh. I'm no fan of the C++ approach, but I'd still rather see C++ follow through on its strategy than half-assing it and becoming a watered-down copy of Rust. People who want Rust know where to find it.
Historically correctness wasn't seen as an important goal in C++ and so no, I don't think any of the three popular C++ stdlib implementations will do something vaguely reasonable for nonsense sort input. It's potentially faster (though bad for correctness) to just trust that this case can't happen since the programmer was required to use types with total ordering. So yes, I'd expect it to result in bounds misses in real software.
I wouldn't think you could get bounds misses without doing extra comparisons, so I'd expect real-world sort implementations to just fail to sort, which doesn't seem particularly unreasonable. But in any case, it's a huge leap from "existing C++ stdlib implementations behave badly in this case" to "the C++ standard requires every implementation to behave badly in this case".
Can you give an example of non-C++ code that a modern compiler (MSVC, clang, g++ or something) successfully compiles with no diagnostics? I’m genuinely curious. If not, this just sounds like more C++ FUD because the spec doesn’t define everything under the sun and allows a certain amount of leeway to compilers for things like emitting different error diagnostics.
Per the C++ standard ([lex.name]/3), this program is ill-formed:
> In addition, some identifiers appearing as a token or preprocessing-token are reserved for use by C++ implementations and shall not be used otherwise; no diagnostic is required. [...] Each identifier that contains a double underscore __ or begins with an underscore followed by an uppercase letter is reserved to the implementation for any use.
Thus, the compiler theoretically has the liberty to emit whatever it wants for this program.
Neither GCC nor Clang produces a warning under -std=c++20 -Wall -Wextra (Clang only produces a -Wreserved-macro-identifier under -Weverything), and MSVC doesn't produce a warning under /std:c++20 /Wall.
In practice, most examples of ill-formed programs where compilers issue no warnings occur with discrepancies between different source files that are linked together; e.g., declaring a function as inline in one file but non-inline in another, or declaring a function with two different sets of default arguments in different files, or defining the same non-inline variable or function in different files, or defining the same inline function differently in different files.
Don't forget since C++ 20 all programs which rely on a concept but don't fulfill the (unchecked) semantic constraints of that concept are ill-formed. This allows C++ to truthfully say that C++ programs have the same properties as Rust programs in this regard, because although most real world C++ may fail those constraints and they aren't checked, as a result those aren't technically C++ programs at all so the fact they're broken and don't do what their programmers expected is magically not the fault of C++ even though the compiler seemed fine with it.
> Thus, the compiler theoretically has the liberty to emit whatever it wants for this program.
In theory, sure. In practice, what does it do? We can look and see.[0][1][2] (I don't know of any compiler that emits garbage in the example you listed). For some reason, there's this sentiment that's arisen that treats undefined behavior as some sort of bugaboo that's capable of anything and everything (including summoning nasal demons).
Here's a question that I can't find an answer to. Is machine code well-defined (as in, can it contain undefined behavior)? If the answer is yes, then all Rust programs can also contain undefined behavior, because they eventually turn into machine code after all. If the answer is no, then that means once a C++ program is compiled into machine code, the executable is a well-defined program.
This whole nasal demon garbage was a good hyperbole at the time to explain that, yes, undefined behavior is bad. But it's been taken to this extreme where people will use arguments like this to try and convince others that if your program has undefined behavior, then it could summon a nasal demon, because who knows what could happen.
In reality, that won't happen. In reality, a signed integer overflow can result in a meaningless program, or a security vulnerability, or many other bad things, but it doesn't mean that the compiled machine code is all of a sudden inventing new instructions at random that the CPU will happily accept. It doesn't mean that your program will turn your OS into a pac-man game. It doesn't mean that it's impossible to find the root cause of the issue and remove the undefined behavior.
In practice, undefined behavior means that your program is broken, but you can look at the compiled program and trace exactly what it does. Computers are machines, they behave predictably (for the most part). They fetch an instruction, execute the instruction, and read/store results to memory. If you look at the instruction and memory (the cache stored on the specific core that the processor is using), you can reliably predict what the CPU will do after executing that instruction.
And yes, multithreading exacerbates the effects undefined behavior and makes it more difficult to debug. But you can debug the issue, even if it means you have to look at the machine code directly.
So while, yes, a compiler can emit garbage when it encounters the example program you gave, using that as an argument for why undefined behavior is bad is dumb. Because in reality, compilers will do something that makes sense, and if it doesn't, you can just look at what it produced. In all the examples you listed, I'm sure that if you created a small example illustrating those situations, the compiled program would do what you expect it to. (Like in the case of having different default args, it will probably just throw an error like this[4]).
Sorry for the rant, it just annoys me that we can't discuss undefined behavior without somebody making an argument that doesn't matter in virtually every case. Undefined behavior is bad! But there are good, real, common examples that show why it's bad! Use those instead instead of talking about how compilers are technically permitted to spew trash when they encounter a program that looks normal, because I'd be very surprised if any modern compiler exists that does spew trash for a normal looking program.
This talk by Chandler Carruth seems pretty good on explaining the nuance of undefined behavior[5].
First of all, and most importantly, we're not talking about Undefined Behaviour, which happens at runtime, but about IFNDR (Ill-formed, No diagnostic required), which means at compile time your program has no meaning whatsoever because it's not a well-formed C++ program after all but your compiler doesn't tell you (and because of Rice's Theorem in many cases cannot possibly do so as it can't determine for sure itself)
This is a conscious choice that C++ made, and my belief is that once you make this choice you have an incentive to make it much worse over time e.g. in C++ 11, C++ 14, C++ 17, C++ 20, and now C++ 23. Specifically, you have an incentive to allow more and more dubious code under the same rule, since best case it works and worst case it's not your fault because if it doesn't work it was never actually a C++ program anyway...
Still, since you decided to talk about Undefined Behaviour, which is merely unconstrained misbehaviour at runtime, let's address that too.
For the concurrency case no, humans can't usefully reason about the behaviour of non-trivial programs which lack Sequential Consistency - which is extremely likely under Undefined Behaviour. It hurts your head to even think about it. Imagine watching an avant garde time travel movie, in which a dozen characters are portrayed by the same actor, the scenes aren't shown in any particular order and it's suggested that some of them might be dreams, or implanted memories. What actually happened? To who? And why? Maybe the screenwriters knew but you've no idea after watching their movie.
Today a huge proportion of software is processing arbitrary outside data, often as a service directly connected to the Internet. As a result, under Undefined Behaviour the unconstrained consequences may be dictated by hostile third parties. UB results in things like Remote Code Execution, and so yes, if they wanted to the people attacking your system certainly could turn it into a Pac man game.
We should strive to engineer correct software. That we're still here in 2023 with people arguing that it's basically fine if their software is not only incorrect, but their tools deliberately can't tell because that was easier is lunacy.
> First of all, and most importantly, we're not talking about Undefined Behaviour, which happens at runtime, but about IFNDR
From the link I posted, from cpp reference, which gives a definition for what constitutes undefined behavior:
>> ill-formed, no diagnostic required - the program has semantic errors which may not be diagnosable in general case… The behavior is undefined if such program is executed
And as for the rest of your argument, you still haven’t answered the one question that matters. Is an executable file with machine code well-defined? If it is, then once you compile a C++ program, the generated binary is well-defined. And once again, all the superfluous arguments about what could happen are irrelevant when there’s no context provided. There’s lots of different types of undefined behavior. An integer overflow by itself does not make your program susceptible to remote code execution. It’s the fact that the integer overflow gets stored in a register that later gets used to write to invalid memory, and then that invalid memory has harmful code injected into it that gets executed.
We should strive to engineer correct software. I agree. It’s all this hand waving about how all undefined behavior is equal that irks me. Because it’s not true, as evidenced by the example listed above about the program that’s “technically” ill-formed C++, but in practice compiles to a harmless program. Engineering requires an understanding of what could go wrong and why. Blindly fretting about all undefined behavior is useless and doesn’t lead to any sort of productive debates.
Engineering is all about understanding tradeoffs. Which is one of the reasons the C++ spec does not define everything in the first place. The fact that in 2023 people don’t understand that all decisions in software come with a tradeoff is lunacy. And once again, I agree that undefined behavior is bad, but to pretend that the existence of IFNDR means the whole language is unusable is silly. People use C++ everyday, and have been for almost 40 years. I’d like more productive conversations on how rust protects the user from the undefined behavior that makes an impact, not about how it doesn’t have IFNDR because that’s irrelevant to the conversation entirely.
> the one question that matters. Is an executable file with machine code well-defined?
While you've insisted that's somehow the one thing which matters I don't agree at all. Are programmers getting paid to produce "any executable file with machine code" ? No. Their employer wanted specific executables which do something in particular.
And there aren't "different types of undefined behavior" there's just one Undefined Behaviour. Maybe you've confused it with unspecified behaviour ?
In the integer overflow case, because that's UB (in C++) it's very common for the result to be machine code which entirely elides parts which could only happen under overflow conditions. Because overflow is UB that elision didn't change the program meaning and yet it makes the code smaller so that's a win - it didn't mean anything before in this case and it still doesn't - but of course the effect may be very surprising to someone like you.
The excuse that "We've done it for decades therefore it can't be a bad idea" shouldn't pass the laugh test. Did you notice you can't buy new lead paint any more? Asbestos pads for ironing boards? Radium dial wristwatches? "That's a bad idea, we shouldn't do that any more" is normal and the reluctance from C++ programmers in particular shows that they're badly out of touch.
This is a common refrain in C++ testing: if it compiles then it's probably correct.
> Rust has accounted for so many errors that many common test cases become irrelevant
In practice, if you think this way I think it's a sign that you aren't testing the right things in those other languages. You should be testing business logic, not language stuff.
If you look at your test code and think "I would test this in JavaScript, but I don't need to do that in Rust" then just delete the test.