Interesting research, but it is still fascinates me why AI devs of current SOTAs ignore possibility to add numbers as first-grade citizens to AI. like for example suggested here: https://huggingface.co/papers/2502.09741
clean separation matter, it’s really strange to force models to mimic numbers and math via incredibly unfit token-mangling stuff, imho
They definitely does not aware of soviet reality that “roof over head” usually is not in the place where human want to live, same with job. if student after university decided (not by student, by state distributing workforce) to go work at city on polar circle - that means that student will go live and work here, without sunlight for the rest of his life! not joking, personal story with soviet collapse as happy ending (moved to normal place after that)
regarding “math with tokens”: There was paper with tokenization that has specific tokens for int numbers, where token value = number. model learned to work with numbers as numbers and with tokens for everything else... it was good at math. can’t find a link, was on hugginface papers
Shouldn't production models already do this? They already tend to use tokenizers with complex rules to deal with a lot of input that would otherwise be tokenized in a suboptimal way. I recall a bug in an inference engine (maybe llama.cpp?) because of an implementation difference in their regex engine compared to the model trainer. Which means that the tokenizer used regex-based rules to chop up the input.
in paper mentioned “number” is a single sort-of “token” with numeric value, so network dealing with numbers like real numbers, separately from char representation. All the math happens directly on “number value”. In majority of current models numbers are handled like sequences of chars
clean separation matter, it’s really strange to force models to mimic numbers and math via incredibly unfit token-mangling stuff, imho