they would have access to the socket not the key, sure a very elaborated attack can probably figure out how to exfiltrate a lot of things (since they have already compromised the host) but for most, if they don't see things in ~/.ssh they would just go away and figure out another host to exfiltrate keys