I thought it had been common knowledge that the way handwritten mail is sorted is that it is barcoded and photographed, and then the address is keyed on by a human.
It doesn't take much imagination that these days each such address is kept, and that the three-letter agencies have a sample of everyone's handwriting, in case it is needed.
Only if the OCR couldn't process it, although the OCR has gotten a lot better so the "Remote Encoding Centers" see a lot less volume. You could always tell when your mail failed the OCR in the past because it would have a faint orange barcode printed on it -- this was used to track it through the human encoding system at the RECs.
It doesn't take much imagination that these days each such address is kept, and that the three-letter agencies have a sample of everyone's handwriting, in case it is needed.