Thanks, I now found a wiki describing these mechanics, and it turns out that UO had an entire system of markings for players who committed illegal activities. This clears things up a bit.

Idk where the ‘killing the player’ part came from—though local game journalism at that time was quite, uh, idiosyncratic. But, as the workings of these guards were somewhat hazy in my mind, they gained an ominous air.

I think I know why you have this hazy memory! Oftentimes these robber groups patrolled around the edges of the "safe zones" where it's possible to call guards. When there were battles, it was often chaotic with multiple characters moving and mauling each other. Sometimes an innocent player would be attacked near the border, they would defend themselves and accidentally do some action which marks them as a criminal (for example, moving inside the border of the safe zone and defend against an attacker who is outside the safe zone and not marked as criminal). The innocent player defending themselves might call "guards", but because they've accidentally made some particular action, the guards would treat them as the criminal and kill them.

Though at the beginning the guards didn't teleport and instakill you, instead they would have to chase you - I don't remember if their hit was an instakill but I do remember a before/after of when suddenly you could not escape the guards.

Originally they were just standard NPCs. They did not instakill. And while quite tough, they could be defeated. So a strong enough group could go in and clear out the guards until they respawned. At which point the town became lawless. I think the shift towards effectively being deities smiting criminals was complete by the end of Beta test though.

As an avid player I can confirm that all commands would execute. So if you were next to a vendor, you'd see the buy menu, but also if a criminal was nearby they'd have the guards summoned on them at the same time.