There is no generic algorithm to completely prevent cancelation. However, there are a lot of specific little ways you can do algebra to push it around so it doesn't hurt you badly (note that I said "avoid", not "prevent"). I would conjecture that the vast majority of numerical systems can be designed that way if you take the time to think about it.
>I would conjecture that the vast majority of numerical systems can be designed that way if you take the time to think about it.
Sometimes there are ways to mitigate this, sometimes there aren't. Sometimes you need to precondition, sometimes you need to rearrange, sometimes you need a different algorithm, sometimes you need to normalize, sometimes you need to use a different arithmetic and so on.
For solving linear systems alone, there are definitely thousands of papers dealing with the problems arising from this. For every single algorithm you write and for all data which comes into that algorithm, you need a careful analysis if you want to exclude the potential of significant numerical errors.
Your comment makes it seem like this is a small problem, where you can just look at an algorithm for a time and fix it, this is literally a hundred year research project in numerics.
> Sometimes there are ways to mitigate this, sometimes there aren't. Sometimes you need to precondition, sometimes you need to rearrange, sometimes you need a different algorithm, sometimes you need to normalize, sometimes you need to use a different arithmetic and so on.
> For solving linear systems alone, there are definitely thousands of papers dealing with the problems arising from this. For every single algorithm you write and for all data which comes into that algorithm, you need a careful analysis if you want to exclude the potential of significant numerical errors.
It sounds like we agree that cancellation is avoidable with some analysis, and there are hundreds of techniques you can use to deal with it, but mostly it's the ~5 you listed there. And as you suggest, I don't believe this is nearly as significant a problem in the general case as you think it is. A careful error analysis is possible if you care (and if ever you cared, it would be on a spacecraft), and far easier in floating point than in many other number systems, including fixed point number systems.
Numeric systems that truly fix cancellation are incredibly big and heavy, and cannot usually be used for real-time calculations in a generic form. Fixed point certainly doesn't fix cancellation - it introduces precision loss issues on every operation you do that causes a number to go down in magnitude. It is actually harder to design systems in fixed point that avoid massive precision losses than it is in floating point, and the error analysis is much more substantial.
>I don't believe this is nearly as significant a problem in the general case as you think it is.
My original comment was about manned space flight in particular. If your application is relatively generic I think it is completely okay, if you are aware of it and mitigate the most pressing issues.
>Numeric systems that truly fix cancellation are incredibly big and heavy, and cannot usually be used for real-time calculations in a generic form.
You can use interval arithmetic, which guarantees that you at least know when cancellation has occurred. Interval arithmetic is fast enough for real time, although it has its own significant drawbacks.
> It is actually harder to design systems in fixed point that avoid massive precision losses than it is in floating point, and the error analysis is much more substantial.
Absolutely. My point was, that a manned space craft, might just be the point to do it.
> My original comment was about manned space flight in particular. If your application is relatively generic I think it is completely okay, if you are aware of it and mitigate the most pressing issues.
Everything we have been talking about relates to space flight. In fact, with humans on board, you can afford to be a lot less precise, because they can work around most numerical issues by hand. The Apollo guidance computers, for example, were prone to occasional instances of gimbal lock and numerical instability, and the astronauts just fixed it.
> You can use interval arithmetic, which guarantees that you at least know when cancellation has occurred. Interval arithmetic is fast enough for real time, although it has its own significant drawbacks.
Interval arithmetic does not prevent cancellation. It's just two floating point calculations, both of which are actually less precise than the one you would do otherwise (you don't use default rounding for interval arithmetic, you round the bottom down and the top up). You do know when things have been canceled, but you know that in a floating point calculation anyway if you have done the error analysis.
My overall point here is that NASA isn't missing anything by using floating point instead of using other weird or exotic arithmetic systems. Double-precision floating point combined with a rudimentary error analysis and some algebra is good enough for pretty much everything, and you may not be able to do better at all with fixed point. Designing fixed point algorithms also depends on a very careful analysis of interval ranges and precisions, and often gets you nothing over just using "double", where the error analysis is easier anyway.
If you need to do better than double, there's also double-double arithmetic for your hard parts, which is a similar speed to interval arithmetic and doubles the precision you get beyond double.
Or you can just use something like herbie that thinks about it for you: https://herbie.uwplse.org/