Limits are not an inherent part of calculus. You can do all calculus relevant for the physical world just fine with nilpotent infinitesimals if you but give up excluded middle.
I've heard of this constructivist approach to calculus, but hadn't made the connection with nilpotents. that's really interesting, could you explain why nilpotenxy and forgoing the law of the excluded middle relate to each other?
You can use nilpotents with classical logic and the excluded middle. This is called dual numbers and it's already a good model for "calculus without limits". They are like complex numbers, but instead of x^2=-1 you set x^2=0.
However, if you want to get really serious about that, you'll need that zero plus an infinitessimal be equal to zero. This is impossible in classical logic due to the excluded middle (which forces each number to be either equal to zero or non-zero).
The Silvanus P. Thompson book suggested by the sibling comment is lovely and very clear.
For a more algebraic treatment, and its important applications to automatic differentiation, I'd suggest starting with the relevant wikipedia articles:
My point is not that limits are an inherent part of calculus. My point is that calculus as currently taught mixes infinitesimal-like notation that predates limits with limit-based calculus.
Statistics is even worse. A mix of old tricks developed to avoid computations when these were expensive. See [2].
[1] A Radical Approach to Real Analysis https://www.davidbressoud.org/aratra/
[2] The Introductory Statistics Course: A Ptolemaic Curriculum? https://escholarship.org/uc/item/6hb3k0nz