On modern AWS instance types so much is offloaded to dedicated hardware that the only shared (noisy) components between VMs is memory bandwidth and higher levels of CPU cache (and I think graviton doesn't even share CPU cache now)
I would suspect your performance difference is mostly likely showing that on metal you are sharing the same software wider so not polluting caches as much as a vm neighbour running unrelated software.
The virtual instance is the exact same size as the metal one, which covers an entire physical machine - I guess this is pure overhead rather than noisy neighbors.
I would suspect your performance difference is mostly likely showing that on metal you are sharing the same software wider so not polluting caches as much as a vm neighbour running unrelated software.