What do you mean by “trust cloud trail”

So cloud trail shows the compromised account logging into an EC2 instance every day like normal.

Then service account credentials are used to access user data in S3.

How does cloud trail indicate the compromised credentials were used to access the customer data in S3?

If you have data events enabled for your S3 bucket, CloudTrail will log every access to that bucket along with the identity of the principal used to access it. https://docs.aws.amazon.com/awscloudtrail/latest/userguide/l...

Right and in my example it would be the principal of the service account, not the compromised AWS account.

If you ran a cloud trail query that's essentially "Did Alice access user data in S3 ever?" the answer would be "No"

So that brings us back to the question, what is meant by "trust CloudTrail"

Most non-trivial security investigations involve building chains of events. If SSM Session Manager was used to access the EC2 instance (as is best practice) using stolen credentials, then the investigation would connect access to the instance to the use of instance credentials to access the S3 bucket, as both events would be recorded by CloudTrail.

CloudTrail has what it has. It's not going to record accesses to EC2 instances via SSH because AWS service APIs aren't used. (That's one of the reasons why using Session Manager is recommended over SSH.) But that doesn't mean CloudTrail isn't trustworthy; it just means it's not omniscient.