A lot of commenters suggesting "pipefail" aren't realising the full extent of the problem. Here's an example that might be clearer, from https://david.rothlis.net/shell-set-e
This prints “a”, as you’d expect:
set -e
myfun() { printf a; false; printf b; }
myfun
printf c
...because the “set -e” terminates the whole script immediately after running the “false” command.
But this script prints “a-b-True”, where some people (myself included) might have expected it to print “a-False”:
set -e
myfun() { printf a-; false; printf b-; }
if myfun; then
printf True
else
printf False
fi
The “set -e” is ignored completely. Putting another “set -e” inside the definition of myfun doesn’t make any difference.
It's always been odd that Bash has hundreds of options for controlling its behaviour in non-standard ways, via "set", "shopt" and environment variables, but it's never had an option to make "-e" do the obviously natural behaviour,,
I encountered the "you can't enable -e" problem years ago in a complex build system that used Bash to build embedded sysem components. The scripts were all designed on the assumption that "set -eu -o pipefail" would cause early exit on any failing commands as if doing the equivalent of "exit $?" or "return $?", and of course it is not reliable, as your example shows.
The very surprising behaviour of "set -e" inside functions was not noticed for a long time while the system was being written and used. When it was noticed, "set -e" was added inside the functions because that seemed to work for the Bash version at the time, then later it stopped working.
it's never had an option to make "-e" do the obviously natural behaviour,,
This is harder than it sounds! :) There's arguably not a natural behavior for shell, because $? is overloaded in Unix. It can mean success/failure or it can mean true/false/error.
Oil introduces some new idioms that led you distinguish between the two paradigms, so it's best to change the way you write scripts. It's not something bash can magically fix.
Here's a demo of how Oil fixes it, which I mentioned in this thread:
$ osh hn.sh
a-b-True
You can add shopt --set strict_errexit at the top of your script:
$ osh -o strict_errexit hn.sh
if myfun; then
^~
hn.sh:3: errexit was disabled for this construct
if myfun; then
^~~~~
hn.sh:3: fatal: Can't run a proc while errexit is disabled. Use 'try' or wrap it in a process with $0 myproc
Or just use Oil:
$ oil hn.sh
<same output>
So Oil is acting like a ShellCheck at runtime, to unconditionally flag this error. It cannot be caught using syntax (which ShellCheck is limited to), because function vs. external executable is a runtime decision.
In the 2000s, I was running extensive sets of simulations and data reduction scripts for a scientific experiment, and I was heavy relying on scripts to run the programs, collect the results, and distribute them over several servers. At first I developed those scripts using bash, but I needed to do math and complex iterations over file names and different parameter files, and I continuously stumbled upon weird behaviors and had to rely to hard-to-understand quirks like the ones explained in the article (which bit me more than once!).
After a while I stumbled upon scsh [1], which at first didn't impress me because I ran it as an interactive shell, and from this point of view it was really ugly. But then I realized that scsh was primarily meant as a way to run shell scripts, and I immediately felt in love. I had the power of a Scheme interpreter, the ability to easily use mathematical expressions (the awesomeness of Scheme's numerical tower!) and macros, and a very effective way to redirect inputs and outputs and pipe commands that was embedded in the language [2]!
In those years I used scsh a lot and developed quite complex scripts using it, it was really a godsend. Unfortunately the program got abandoned around 2006 because it was not trivial to add support for 64-bit architectures. However, while writing this post I've just discovered that somebody has revived the project and enabled 64-bit compilation [3]. I would love to see a revamp! Nowadays I use Python for complex scripts, but it's not the same as a language with native support for redirection and pipes!
Whenever I need a quick script I use bash. If that script goes beyond 20 lines or has anything but the simplest `if` or `&&`, I've learned time and time again that I should rewrite it (python for example)
> Nowadays I use Python for complex scripts, but it's not the same as a language with native support for redirection and pipes!
While this is no doubt right, I think you could build a little abstraction that simplified piping syntax considerable. The subprocess module is not terribly convenient, although it has gotten less confusing over the years.
Python does have operator overloading, so you could even make it fancy.
> However, while writing this post I've just discovered that somebody has revived the project and enabled 64-bit compilation [3].
That is excellent news, if it comes to anything. At present it looks like it's still dependent on the 32-bit Scheme48. Is there a roadmap for this project?
> Nowadays I use Python for complex scripts, but it's not the same as a language with native support for redirection and pipes!
Note that Guile has support for both these. But it doesn't have the shell-tool minilanguages in it that scsh does, like the one for awk.
I had a very similar experience with using scsh for shell scripts instead of bash and it was great and far more convenient than using Python or any other language with poor support for the shell style of command execution.
I have also been forced to revert to bash later so it is good to know that there might be an update for it.
For simplish parallel runner https://www.gnu.org/software/parallel/ is excellent, can even automate stuff like "copy a file to remote server then run command on it" so you can make simplistic cluster computing with it
It runs your bash scripts but you can also opt into correct error handling. The simple invariant is that it doesn't lose an exit code, and non-zero is fatal by default.
I spent a long time on that, but the post didn't get read much. I think it's because it takes a lot of bash experience to even understand what the problem is.
Never used OilShell (OSH) before, but this opening line on their home page struck me:
> … and it's a new language for Python and JavaScript users who avoid shell!
Why not work towards using python/JavaScript as shell languages? There was a HN thread a few days ago on Xonsh, the main python attempt at this which looks nice to me, which naturally got a lot of anti-anti-bash energy. But if we want better shell ergonomics and are willing to ditch bash, why not take popular versatile and capable languages and get them shell-appropriate? It seems like a natural use case for these languages, and even the place where they could retire to peacefully in the future.
I suppose OSH is aiming at bash compatibility, but how good or viable is that really (not a rhetorical question)? Seems to me like an all or nothing thing as someone who recently ran into some bash-zsh incompatibility.
Note that osh is (really very nearly[1]) bash compatible by default, but you can opt in to incompatible parts. This lets you convert a large body of shell code gradually.
1: I think the only incompatibility is that in osh you can't index into associative arrays with unquoted words. IIRC this was to make parsing of osh decidable.
I tried xonsh for a while but it was just so slow to chdir a good fraction of the time that I gave up. (Running under WSL2, navigating around linux and windows directories.) Also looking up the syntax was awkward and I still had to copy and paste to make editing multi-line commands be sane.
Nice run down. But in this context it seems to boil down to
> I have the feeling that trying to address these points by bolting features or DSLs onto a general-purpose programming language doesn't work. At least, I have yet to see a convincing implementation of it.
… and what is worth doing, a new shell or new features in an old language, which isn’t addressed by that post AFAICT.
A short answer would be that dozens of projects like this already exist and have existed for decades (scsh was mentioned in this thread), and they aren't widely used as shell replacements:
They're separate things because introducing MORE error handling can make your script incompatible with bash, for people who want to write code that's portable to 2 shells :-/
Mainly, the reason why Bash's "set -e" doesn't do what you expected is this.
set -e comes from the POSIX shell language, descended from the Bourne Shell.
The examples you're using use obscure Bash features. For instance let turns an arithmetic result into a termination status, where 0 is fail (opposite to the POSIX convention and all).
set -e works to the extent that your commands have a sane termination status, which they generally do if they are standard built-ins or well-behaved utilities.
One Bash feature improves the effectiveness of set -e (or exit status testing in general). In a command pipe:
a | b | .. | z
the termination status is obtained from z. That's standard. If z indicates success, the pipeline is successful no matter how a through y terminate. Bash has a "pipefail" option to help with this.
I moved over to shellharden a while ago. It can actually apply the suggestions it makes. Aside from that, my employer is somewhat disapproving for GPLv3 tools, but MPL that shellharden uses is essentially auto-approved.
I tend to think of Bash as a syntax-free language. There are no clear rules, you just have to kind of make it work every time. It's like the programming version of a freestyle rap or Parkour.
It's more of a pidgin. Syntax and semantics made up on the fly to suit the need of the moment, and then made permanent once used. No overarching design or consistency.
You don't have to be hyper-aware of false positives with "set -euo pipefail". False positives bring themselves to your attention during testing, while false negatives don't announce themselves at all. It's almost always preferable for your code to incorrectly fail and force you to stick "|| true" on the end, than to incorrectly succeed and let you miss the bug.
There are false negatives, such as the one mentioned in the post after “A particularly dangerous pitfall with set -e is combining functions with conditionals.”
All set -e does is halt further execution of your script if any line exits above 0. There is really nothing bash can do about this. It can't perform a psychological evaluation on why a program is not giving the expected output. And most of the time, when something fails, if it were made by a less than serious programmer (like myself) they don't bother to exit on error correctly with a code above zero. But if you are building a script collection, or python utility or even an binary, simply exit 1 if you encounter a general error. Then further up the chain in your shell scripts the error can be handled properly. There are more specific error codes you can use that might be helpful to your shell script.
If you are wondering what in the world I'm talking about and I've missed your question entirely, sorry about that. My feet are tired.
The `let ++` example is pretty damning. I find Bash to be a poor and outdated shell that we’re stuck with for now. Its design is the best argument against LSD I know, it came out of the hippie drug days, and Bash reflects that with its madness.
A lot of the sins of shell are from it's primary role as an interactive interface to the computer. it's use as a scripting language was a nice secondary goal.
On the one hand it is nice to have your interactive interface and your scripts be the same language. on the other it is a bit horrifying the way all those convenient interactive features make your programs so prone to failure.
We'd probably all be better off if our scripts weren't in the same language as used by our interactive interface. I do paste snippets of scripting (if, for, etc.) into my interactive terminal from time to time, but that's rare compared to just typing commands to run.
I wouldn't mind it if all scripting in the shell had to be offloaded to a completely separate language that was less error prone and more consistent.
I don't mean what it returns, I mean that returning non-zero trips up Bash into thinking it's a non-successful error code. The fact that doing math can trip up error detection implies a fundamental design flaw of the system. Which, I mean, we're in Bash, of course it's poorly designed. I didn't know it was quite this bad though.
There's neither anything disingenuous about my claim. Bash's behavior is quite unique and unexpected to most people in the examples I showed. And I don't appreciate the attack.
Yep as a programing language shell is weird, I am sure you know this better than me, but I got nerd sniped by your snippet, and want say what I think is happening.
false the command returns 1,
! the command turns a 0 return code into a 1 and anything other than a zero into a 0,
if is vaguely backwards from most languages in that it deals in return codes a rc of 0 is true and anything else is false
in the first case false has a rc of 1 this will execute the else list then the test command 1 is equal to 0(rc = 1 false list) so "msg" does not print
in the second case false(rc 1), !(rc 0), if(true list) now you check if rc is 0, it is and "msg" is printed.
Yeah. Basically instead of treating ! specially, Bash treats it like any other command that sets $? afterwards. And since it always succeeds, you always end up with $?=0 after it.
I don't know what kind of shell scripts the designers of this write, but the number of times I've wanted or found it helpful for ! to modify $? is has been exactly zero.
I am almost certain that >99% of people who write shell scripts do not find this intuitive.
> Yeah. Basically instead of treating ! specially, Bash treats it like any other command that sets $? afterwards. And since it always succeeds, you always end up with $?=0 after it.
I don't think that is true, consider:
# if false ; then echo "true $?" ; else echo "false $?" ; fi
false 1
# if ! false ; then echo "true $?" ; else echo "false $?" ; fi
true 0
# if ! ! false ; then echo "true $?" ; else echo "false $?" ; fi
false 1
Basically, $? evaluates to the full condition inside the "if" sentence.
One of the first things I learned when starting to use $? was to assign it to a variable asap if I needed to use it more than once.
Whoops, that's what I get for writing these when I'm sleepy. Yes of course ! doesn't always set the result to success, that wouldn't make any sense. I misspoke in that part. I meant to say Bash treats it like any other command that modifies $?, instead of treating it specially and preventing it from modifying that variable. Thanks.
Shell scripts can be thought like C++. They can be sanely managed only if one adopts a strict subset of functionalities.
The page is probably oriented at situations where one needs to support any version of bash and any obscure/inadvisable functionality (mind that there are still devs that mix sh and bash syntax in the same script), which is very inconvenient and error prone, so manual error handling may make sense.
While there is always something insane behind the corner in Bash, with a restricted subdomain (e.g. bash 4.2+, strict shell options, and shellcheck) it's possible to progressively write reasonably solid shell scripts.
The document conclusion is somewhat biased. "to handle errors properly" implies that `-e` in inherently unreliable, which not fair - strict shell options do remove certain classes of errors, which doesn't hurt.
C++ can be used correctly. It's easier since C++11, with smart pointers and stuff. I'm not going to argue that writing solid C++ is easy, but it IS possible.
Bash basically can't be used correctly. Even a simple command pipeline of three commands, with actual error handling, is probably more code in bash than in any other language.
> I disagree. Bash basically cannot be used correctly. It's like gets() in C; a knife with a blade but no handle. Unsafe at any speed.
> C++ can be used correctly. It's easier since C++11, with smart pointers and stuff. I'm not going to argue that writing solid C++ is easy, but it IS possible.
It seems this conflates two different aspects of programming languages - complexity and (memory) safety.
The analogy with Bash was in relation to the language complexity. By reducing the language features to a restricted, well-known (by the given team) scope, it's possible to use the language effectively. A very good example of this is the Doom 3 source code.
In relation to memory safety, Google hasn't been able to develop a safe enough Bluetooth stack, which has been a dumpster fire. Due to the small support windows of Android, as of today, a huge amount of devices with older Android versions are either in the garbage, or are extremely unsafe. If Google, with their engineering practices, can't develop a safe Bluetooth stack in an unsafe language, realistically, nobody can.
The topic in the post is very interesting (pipes error handling are no doubt a rough area of Bash), but the post takes the choice of being a rant rather than an informative research. The central thesis that "functionality of language X is broken in a 1-line program, but it works in a 83-lines program in language Y" doesn't have any meaningful conclusions. It would have been interesting and informative to compare against the correct version in language X, which in the case of Bash, it does exist (but in this case, it's advanced Bash, and sure, it's ugly).
There is a very wide gap been 1-, 10-, 100- and 1000- line programs, and very different requirements. Asserting that only the 1-line case is the only valid use case for shell scripting, it's inefficient system administration (assuming disciplined shell scripting).
> "functionality of language X is broken in a 1-line program, but it works in a 83-lines program in language Y"
Fair criticism. The difference though being that bash is implying a promise of being able to provide a solution in one line, but if you actually try to make bug free programs it'll end up being just as long as in Go. Without the readability, since you have to jump through crazy amounts of hoops since even the fixes/workarounds for the problems in bash don't have support in the language.
This makes bash's supposed strengths actually weaknesses, even for small programs.
> (pipes error handling are no doubt a rough area of Bash)
But that's my point when I condense this to calling bash useless. Pipes are maybe the core way that one ties together components in bash, and error handling is very important to not have bugs. Unhandled or mishandled errors are bugs.
> Asserting that only the 1-line case is the only valid use case for shell scripting
Well, I'd say that a 1-line case will not be correct, but will work in most day to day, if supervised.
And a correct shellscript can never be 1 line, and making it correct would make it 10s or 100s of lines of unreadable mess, that nobody would actually write or maintain.
Maybe there are wizards out there who write bug free bash. I've never met one. I've met many who think they are.
Most would probably be surprised that they'd have to write at least tens of lines of code to replace their use of the pipe operator, to have bug free code.
That was no idle musing; he had in mind a reversal of the situation.
He put that into action by making a language so embroiled in its own implementation that cloning a new version would be a pointless exercise in failing to replicate the internal details of the original program. That nicely solved the problem of it being hard to port a script.
Then he made it incredibly hard to build. Cross-compiling it wasn't even a thing, and probably still isn't; we just use distro builds that have Qemu.
Annoyingly, when a process is terminated by an unhandled signal (say, SIGTERM), it is treated as if it exited with a nonzero exit code. This can make it tricky to use non-builtin commands as conditions in "if" statements, since there's always the potential edge case where the "if" block is skipped because of a signal that the condition received.
I've reviewed a lot of code, bash and otherwise. I have never, not once, reviewed bash code that didn't have subtle bugs. And this is code written by smart people.
It really is absolutely terrible tool as programming language. If your script is more than "a bunch of pipes in a trench coat" just use something else. I'd unironically prefer Perl to it.
Powershell core.
It's actually usable and allows to use .net tools if you need to break out the big guns.
Main drawback is the stupid channel thing, but that's manageable.
> Build scripts that fail and return 0 are my nemesis.
I was also once party to an argument caused by a developer who thought that non-zero exit codes were an acceptable way to communicate success states. I suspect that he may have been trying to communicate different success conditions, but I couldn't tell you for certain.
This is really interesting. The case pointed out in Ex.3 is pretty hilarious:
(from the bash manual)
The ERR trap [same rules as set -e] is not executed if the
failed command is part of the command list immediately following
a while or until keyword, part of the test in an if statement,
part of a command executed in a && or || list except the command
following the final && or || [...]
One of the reasons I created Next Generation Shell. It has exceptions. So "if $(grep ...)" works correctly as opposed to bash. grep exit codes: 0 - found, 1 - not found, 2 - error. bash can not handle this correctly in if. There are just two branches for 3 exit codes. NGS has two branches and exception that can be thrown. Yep, every single "if grep ..." in bash is a bomb.
mountain out of a molehill imo. there’s a certain point where you’ll be fighting shell more than it’s helping—choosing another language is the better choice. i’m not a wizard but i feel like i’ve developed a decent intuition for what’s sane to do in shell and what needs something else.
and even then, shelling out from another language when absolutely necessary can be a better option
IMO, Perl is perfect for this purpose. A good thing about Perl is, it doesn't really changed any more. So no need to spend time keeping up with latest developments like Ruby or Rust.
the `read -r foo < configfile` might be obscure, but every line in a POSIX text file ought to have a newline terminator. If this is worth erroring out on probably depends on context.
I'm not sure what this person did expect. That bash magically parsed the output and memory of running programs, and read the user's thoughts, to determine if the state of the program indicates a condition that the user would consider an error?
set -e does exactly what you'd expect, arguably, with the exception of subshells and conditions.
And those rules are extremely simple to learn too. If you understand when a statement which might be composed of other statements would have an error, you can predict what set -e will do
From help let:
Exit Status:
If the last ARG evaluates to 0, let returns 1; let returns 0 otherwise..
Since var++ is post-increment, I guess the last argument does evaluate to zero. Subtle...
Would you find it less surprising if the let command behaved differently than most other commands, and didn't stop the program if it returned a non-zero exit code?
Because, this example works exactly the way I would expect. The next one (using the double parentheses syntax) is more questionable, and I could see it going either way, and would either look it up in the manual, or put an `|| true` after it to be safe. The fact that it changed in a minor release is pretty bad though.
Not your parent commenter. I'd expect let command not to return a numeric value at all, since bash uses the returned numeric value to indicate success/failure. But that admittedly doesn't have anything to do with the design of "set -e".
This prints “a”, as you’d expect:
...because the “set -e” terminates the whole script immediately after running the “false” command.But this script prints “a-b-True”, where some people (myself included) might have expected it to print “a-False”:
The “set -e” is ignored completely. Putting another “set -e” inside the definition of myfun doesn’t make any difference.