> the whole structure is easily converted to reverse mode afterwards.
Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form".
(Disclosure: I'm the other of another paper on AD.)
I think JAX effectively demonstrates that this is indeed possible. The approach they use is to first linearise the JAXPR and then transpose it, pretty much in the same fashion as the Elliot paper did.
A JAXPR is a normal functional-style expression tree, with free variables, like
let b = f a in g (a, b)
Elliot's "compiling to categories" requires you to translate this to
g ∘ (id × f) ∘ dup
It's pretty baffling to work with terms like the latter in practice! The main ideas that JAX is based on were already published in "Lambda the ultimate backpropagator" in 2008. Elliot's work is a nice way of conceptualising the AD transformations and understanding how they all relate to each other, but it's not particularly practical.
Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form".
(Disclosure: I'm the other of another paper on AD.)