Note that Google Pixel hardware is just fine and not evil, and they're looking at a different vendor for the next version anyway, because Google is making it so the Pixel will only run approved OSes.
The Titan chip does a lot more than sign and store keys. It also has storage (could contain malicious payloads) as well as an RNG and AES/SHA accelerators (which could be weakened/compromised), among other things.