Cloudtrail events should be able to demonstrate WHAT created the EC2s. Off the top of my head I think it's the runinstance event.
replies(3):
Here is what happened:
Some 600 instances were spawned within 3 hours before AWS flagged it off and sent us a health event. There were numerous domains verified and we could see SES quota increase request was made.
We are still investigating the vulnerability at our end. our initial suspect list has 2 suspects. api key or console access where MFA wasn’t enabled.
Based on docs and some of the concerns about this happening to someone else, I would probably start with the following:
1. Check who/what created those EC2s[0] using the console to query: eventSource:ec2.amazonaws.com eventName:RunInstances
2. Based on the userIdentity field, query the following actions.
3. Check if someone manually logged into Console (identity dependent) [1]: eventSource:signin.amazonaws.com userIdentity.type:[Root/IAMUser/AssumedRole/FederatedUser/AWSLambda] eventName:ConsoleLogin
4. Check if someone authenticated against Security Token Service (STS) [2]: eventSource:sts.amazonaws.com eventName:GetSessionToken
5. Check if someone used a valid STS Session to AssumeRole: eventSource:sts.amazonaws.com eventName:AssumeRole userIdentity.arn (or other identifier)
6. Check for any new IAM Roles/Accounts made for persistence: eventSource:iam.amazonaws.com (eventName:CreateUser OR eventName:DeleteUser)
7. Check if any already vulnerable IAM Roles/Accounts modified to be more permissive [3]: eventSource:iam.amazonaws.com (eventName:CreateRole OR eventName:DeleteRole OR eventName:AttachRolePolicy OR eventName:DetachRolePolicy)
8. Check for any access keys made [4][5]: eventSource:iam.amazonaws.com (eventName:CreateAccessKey OR eventName:DeleteAccessKey)
9. Check if any production / persistent EC2s have had their IAMInstanceProfile changed, to allow for a backdoor using EC2 permissions from a webshell/backdoor they could have placed on your public facing infra. [6]
etc. etc.
But if you have had a compromise based on initial investigations, probably worth while getting professional support to do a thorough audit of your environment.
[0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c...
[1] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c...
[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-...
[3] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/s...
[4] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti...
[5] https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-...
[6] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_R...