I agree. It should talk directly to gpg-agent. That way, it would automatically have support for things like the Crypto Stick and smartcard readers, and wouldn't necessarily have direct access to the keys.