Because the core Python team started working with scientific communities very early on, matrix-sig was created in 1995 and led to a bunch of conveniences from e.g. rich indexing (slices and implicit tuples in brackets) to matmul. `table.unpack` and having to convert rich keys to format rich keys into strings is not an upgrade.
And Lua has a great API for calling into C, but for exposing lua APIs from C you’re basically writing bytecode for a stack VM in C. This is not very fun to expose large rich APIs.
An other significant divergence is because Lua primarily targets embedding rather than freestanding, backwards compatibility is pretty far down the list of considerations, you’re not going to do a major lua upgrade once you’ve released your game.
Python had one major compatibility break 5~15 years ago (depending when you switched) and teeth are still gnashing over it, Lua does that much every major version.
Lua may have a better C API, but it's a small and not very useful language.
Python C API is one of the better ones, it's extremely easy to connect C/C++ code to Python.
Python is also extremely good at reshaping data, parsing files, scraping data and so on - the preprocessing part required by many scientific computations.
Remember that Torch originally had a Lua frontend, and was only later turned into PyTorch, citing the same availability of scientific software.
But considering that Lua has a far better C API than Python, why did those initial scientific/mathematical software use Python?