You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing.
The right way to use the go race detector is:
1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod.
2. Have a nightly job that runs unit and integ tests, built with -race, and without caching, and if any races show up there, save the trace and hunt for them. It only works probabilistically for almost all significant real-world code, so you have to keep running it periodically.
3. Accept that you'll have, for any decently sized go project, a chunk of mysterious data-races. The upstream go project has em, most of google's go code has em, you will to. Run your code under a process manager to restart it when it crashes. If your code runs on user's devices, gaslight your users into thinking their ram or processor might be faulty so you don't have to debug races.
4. Rewrite your code in rust, and get something better than the go race detector every time you compile.
The most important of those is 3. If you don't do anything else, do 3 (i.e. run your go code under systemd or k8s with 'restart=always').
> Rewrite your code in rust, and get something better than the go race detector every time you compile.
Congrats, rustc forced you to wrap all your types in Arc<Mutex<_>>, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch.
Don’t get me wrong, Rust has done a ton for safety and pushed other languages to do better. I love probably 50% of Rust. But Rust doesn’t protect against logical races, lovelocks, deadlocks, and so on.
To write concurrent programs that have the same standards of testable, composable, expressive etc as we are expecting with sequential programs is really really difficult. Either we need new languages, frameworks or (best case) design- and architectural patterns that are easy to apply. As far as I’m concerned large scale general purpose concurrent software development is an unsolved problem.
As a sibling said, Go has all the same deadlocks, livelocks, etc you point out that rust doesn't cover, in addition to also having data-races that rust would prevent.
But, also, Go has way worse semantics around various things, like mutexes, making it much more likely deadlocks happen. Like in go, you see all sorts of "mu.Lock(); f(); mu.Unlock()" type code, where if it's called inside an `http.Handler` and 'f' panics, the program's deadlocked forever. In go, panics are the expected way for an http middleware to abort the server ("panic(http.ErrAbortHandler)"). In rust, panics are expected to actually be fatal.
Rust's mutexes also gate "ownership" of the inner object, which make a lot of trivial deadlocks compiler errors, while go makes it absolutely trivial to forget a "mu.Unlock" in a specific codepath and call 'Lock' twice in a case rust's ownership rules would have caught.
In practice, for similarly sized codebases and similarly experienced engineers, I see only a tiny fraction of deadlocks in concurrent rust code when compared to concurrent go code, so like regardless that it's an "unsolved problem", it's clear that in reality, there's something that's at least sorta working.
There are dozens and dozens throughout the stdlib and other popular go code.
The singleflight case is quite common, if you have:
mu.Lock()
if something() {
mu.Unlock()
moreWorkThatDoesntWantMutex()
return
}
mu.Unlock()
most gophers use manual lock/unlocks to be able to unlock early in an 'if' before a 'return', and that comes up often enough that it really does happen.
I see manual lock/unlock all the time, and semi-regularly run into deadlocks caused by it. Maybe you don't use any third-party open source libraries, in which case, good for you congrats.
Maybe it's stockholm syndrome, but the "manual" unlocks in bite sized funcs don't bother me -- it's easy to see that it's going to be unlocked by the time the function returns.
At a minimum now you’ve also got to scan for nil dereferences. Maybe that simple code between the mutex is pulled out into a function in a later commit. Then the function body is changed in the next commit to do something less simple.
This is all depressingly reminiscent of listening to die-hard C enthusiasts insist that avoiding use-after-free bugs and out of bounds reads and other issues are easy as long as you write code perfectly all the time. It’s easy as long as you just always do the right thing.
It’s okay to admit your language of choice has weaknesses. You don’t have to stop using go just because you can acknowledge that data races can creep in if you’re not diligent enough.
I'm confused. I was responding to a complaint about "manual unlocks" (vs. the "certainty" of a Lock(); defer Unlock().
The examples pointed to in the stdlib to complain about that usage pattern could have easily done just that -- but they were optimized to shrink the window of the locking time. My take is that the code paths were simple enough to verify that it was safe to do, and being in the stdlib had both. enough eyes to validate and a driver (being foundational code).
I recognize that Go is far from perfect but think some of the pearl clutching is manufactured (OMG! nil pointers! -- they technically can happen but in my many years of coding in Go it hasn't been a problem of note.)
And of course we should recognize that the locking issue being brought up was a simple one and the story gets worse when talking about complicated concurrency. Then the Go story of "just use the race detector!" is tempered by the fact that it is only best effort and no guarantees that it's found everything (like a bloom filter of certainty).
So year, Go has footguns and other inadequacies (FFI is a bummer). But it's good enough for a lot of things. Rust is on my list to learn, so my only complaint about it is the steep learning curve and I'm an old lazy dev.
Until you have to call a slow function after the mutex access leading to the lock being held long enough to cause problems.
Now you either refactor into multiple functions, while ensuring all copies of possibly shared data when passing function arguments are correctly guarded or ”manually” unlock when you don’t need the mutex access anymore.
OK, but you're not in "Go"-specific problems any more, that's just concurrency issues. There isn't any approach to concurrency that will rigorously prevent programmers from writing code that doesn't progress sufficiently, not even going to the extremes of Erlang or Haskell. Even when there are no locks qua locks to be seen in the system at all I've written code that starved the system for resources by doing things like trying to route too much stuff through one Erlang process.
drop(foo); // Now foo doesn't exist, it was dropped, thus unlocking anything which was kept locked while foo exists
If you feel that the name drop isn't helpful you can write your own function which consumes the guard, it needn't actually "do" anything with it - the whole point is that we moved the guard into this function, so, if the function doesn't return it or store it somewhere it's gone. This is why Destructive Move is the correct semantic and C++ "move" was a mistake.
Yes, though I think tooling could be better; if I had more spare time I'd write a linter which flagged defers in loops that didn't come with an accompanying comment.
I always think, Go is open source why not just fork it to add feature XYZ, then I realize I am better off using languages whose communities appreciate modern language design, instead of wasting my spare time with such things.
If it's solved the solution has been discarded at some point by other developers for being too cumbersome, too much effort, and therefore in violation of some sacred principle of their job needing to be effortless.
> Congrats, rustc forced you to wrap all your types in Arc<Mutex<_>>, and you no longer have data races.
Or you can just avoid shared mutable state, or use channels, or many of the other patterns for avoiding data races in Rust. The fun thing is that you can be sure that no matter what you do, as long as it's not unsafe, it will not cause a data race.
Lock-free and even wait-free approaches are not a panacea. Memory contention is fundamentally expensive with today’s CPU architectures (they lock, even if you ostensibly don’t). High contention lock-free structures routinely perform worse than serialized locking.
The overhead of Mutex for uncontended cases is negligible. If Mutex acquisition starts to measurably limit your production performance, you have options but will probably need to reconsider the use of shared mutable anyway.
Lock-free data structures and algorithms access shared memory via various atomic operations such as compare-and-swap and atomic arithmetic. The throughout of these operations do not scale with the number of CPU cores. Contrary, the throughput usually reduces with the growing number of CPU cores because they need more time for synchronizing local per-CPU caches with the main memory. So, lock-free data structures and algorithms do not scale on systems with big number of CPU cores. It is preferred to use "shared nothing" data structures and algorithms instead, where every CPU core processes its own portion of state, which isn't shared among other CPU cores. In this case the local state can be processed from local per-CPU caches at the speed which exceeds the main memory read/write bandwidth and has smaller access latency.
Lock-free data structures does not guarantee higher throughput. They guarantee lower latency which often comes at the expense of the throughput. A typical approach for implementing a lock-free data structure is to allow one thread to "take over" the execution of another one by repeating parts of its work. It ensures progress of the system, even if one thread isn't being scheduled. This is mainly useful when you have CPUs competing for work running in parallel.
The performance of high-contention code is a really tricky to reason about and depends on a lot of factors. Just replacing a mutex with a lock-free data structure will not magically speed up your code. Eliminating the contention completely is typically much better in general.
This is more fuel for my thesis that every single feature of golang was considered in isolation, and zero thought was put into how any of them would work together.
I’m not sure how else you can explain perfectly idiomatic code (a loop, or a reused err variable, or a closure) causing a program to fall on its face simply by dropping in go’s namesake feature, whose entire purpose was supposed to be that you could simply drop it in.
To actually use `go` you have to do minor contortions like always remembering to copy your loop variables, make new error variables, and also not accidentally capture any external variables in a closure.
Go doesn’t actually help you with any of this, of course. You just have to remember to do it right every single time. None of these things are hard (usually), but the fact that you have to do them at all speaks volumes about the amount of forethought that went into it.
And of course doing all of those steps doesn’t save you if one of the things you tried to copy secretly contains a pointer inside of it, like absolutely everything in golang does. You didn’t know, and it wasn’t even a public member so it wasn’t in the docs. But there was a pointer somewhere deep inside the thing you copied so now you’ve got unguarded concurrent mutation of shared memory.
I disagree there. It is reasonable to run a few service instances with a race detector. I have a few services where _all_ instances are running with it just fine.
I configure ci to run tests with -race and that works out pretty well. I value short ci runs, so testing with -race is a sacrifice for me even if it only adds ~10 seconds typically. I like your idea of a regular job that runs without caching, but your best tip is gaslighting users. Maybe I should start prefixing error messages with “look what you made me do”.
The right way to use the go race detector is:
1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod.
2. Have a nightly job that runs unit and integ tests, built with -race, and without caching, and if any races show up there, save the trace and hunt for them. It only works probabilistically for almost all significant real-world code, so you have to keep running it periodically.
3. Accept that you'll have, for any decently sized go project, a chunk of mysterious data-races. The upstream go project has em, most of google's go code has em, you will to. Run your code under a process manager to restart it when it crashes. If your code runs on user's devices, gaslight your users into thinking their ram or processor might be faulty so you don't have to debug races.
4. Rewrite your code in rust, and get something better than the go race detector every time you compile.
The most important of those is 3. If you don't do anything else, do 3 (i.e. run your go code under systemd or k8s with 'restart=always').