In this pretend program we are using some 3rd party s3 library to fetch some data, some other 3rd party CSV library to extract the data. Now I want to run this on some other remote node. I know I can just sent that sexp to the remote node and have it eval it, but that is only going to work if the right versions of the S3 and CSV library are already in that runtime.
I want to be able to write a function like:
(defun remote-run (prog)
(....))
That can take ANY program and ship it off to some remote node and have that remote node calculate the result and ship the answer back. I don't know of some way in lisp you could ask the runtime to give you a program which includes your calculation and the s3 functions and the csv functions.
In unison, I can just say `(seialize my-program)` and get back a byte array which represents exactly the functions we need to evaluate this closure. When the remote site tries to load those bytes into their runtime, it will either succeed or fail with "more information needed" with a list of additional hashes we need code for, and the two runtimes can recurse through this until the remote side has everything needed to load that value into the runtime so that it can be evaluated to a result.
Then, of course, this opens us up to the ability for us to be smart in places and say "before you try running this, check the cache to see if anyone ran this closure recently and already knows the answer"
In lisp there are symbols and lists. The symbols could be pointers to content addressable hashes, or in-lined content. So in the namespace you could require the library by its content addressable hash, and give it a symbol definition, or just refer to the hash itself.
I'm not saying there exists a Lisp with a turn-key distributed cloud runtime. Like the sibling answer, I'm saying, it's not super complicated. Instead of loading an .so file the regular way, load it via hash.
The car/cdr nature of Lisp makes it almost uniquely suited to distributed runtime IMHO.
As for your last sentence, this touches on compilation. Solve this, and you solve also dependency detection and compilation.
I feel there are so many things which could/should converge at some point in the future. IDE / version control, distributed compute, storage.
An AST (or source) program could have a hash, which is corresponding in a cache to a compiled or JIT-ed version of that compilation unit, and various eval results of that unit etc etc. So many things collapse into one once your started to treat everything as key/value.
Lets say you wrote a imaginary program to sum a column in a csv:
(defun my-program () (let* ((raw-data) (s3-load-file "htpps://...")) ((parsed) (csv-parse raw-data)) ((column1) (csv-column parsed 1)) (mean column1))
In this pretend program we are using some 3rd party s3 library to fetch some data, some other 3rd party CSV library to extract the data. Now I want to run this on some other remote node. I know I can just sent that sexp to the remote node and have it eval it, but that is only going to work if the right versions of the S3 and CSV library are already in that runtime.
I want to be able to write a function like:
(defun remote-run (prog) (....))
That can take ANY program and ship it off to some remote node and have that remote node calculate the result and ship the answer back. I don't know of some way in lisp you could ask the runtime to give you a program which includes your calculation and the s3 functions and the csv functions.
In unison, I can just say `(seialize my-program)` and get back a byte array which represents exactly the functions we need to evaluate this closure. When the remote site tries to load those bytes into their runtime, it will either succeed or fail with "more information needed" with a list of additional hashes we need code for, and the two runtimes can recurse through this until the remote side has everything needed to load that value into the runtime so that it can be evaluated to a result.
Then, of course, this opens us up to the ability for us to be smart in places and say "before you try running this, check the cache to see if anyone ran this closure recently and already knows the answer"