Incident Response

Your Incident Response Playbook Assumes There's a Disk to Image

Jul 6, 20265 min read

The alert fires at 2 AM. CloudTrail logs show an IAM key running reconnaissance against your S3 buckets. The key belongs to a developer who is currently in a different time zone and definitely not running S3 list commands at 2 AM. The incident commander says the phrase that ends every traditional IR engagement: spin up a forensics box and image the affected system.

The problem is that the affected system is a Lambda function. There is no disk to image. The execution environment lasted 200 milliseconds and no longer exists.

This is cloud incident response, and if your team is running a traditional playbook against it, you are spending time on steps that cannot work while the evidence you actually need evaporates.

What the Traditional Playbook Assumes

Classical incident response was built around a set of assumptions that made sense for on-premises infrastructure: there is a physical or virtual machine, it has persistent storage, you can acquire a forensic image of that storage, you can examine memory if you get there fast enough, and you can trace activity through file system artifacts, registry keys, and local logs.

Cloud environments violate most of these assumptions. Compute instances spin up and disappear on autoscaling schedules. Serverless functions execute in environments that last seconds and leave no persistent artifacts. Infrastructure is defined as code and can be reproduced or destroyed in minutes. The system that was involved in an incident may not exist by the time the incident is declared.

What does persist in cloud environments is the API audit trail. That is your crime scene, and effective cloud IR means rebuilding your investigative process around it.

CloudTrail Is the Disk Image

AWS CloudTrail, Azure Activity Log, and GCP Cloud Audit Logs record API calls: which identity made which request, from which IP, at which time, with which result. Every action in a cloud environment is mediated by an API call, and those calls are logged if you have enabled the right logging categories. If you have not, you are investigating without evidence.

CloudTrail management events cover control-plane actions: creating and deleting resources, modifying IAM policies, changing security group rules, launching and terminating instances. Data events cover data-plane actions: GetObject and PutObject on S3, reading from DynamoDB, invoking Lambda functions. Data events are not enabled by default because they generate enormous log volume. The difference between having and not having data event logging is the difference between knowing exactly what data was accessed and knowing nothing about data access at all.

Before an incident happens, turn on data event logging for your critical resources. The cost is real. The investigative gap when it is missing is also real.

The Scope Question Is Different

In a traditional internal network breach, scope starts at the compromised host and expands to lateral movement paths. In a cloud incident, the primary vector is almost always an IAM credential, and the scope question is different: what could this credential do, and what did it do?

An IAM key that appears in a breach report is the beginning of the investigation, not the end. The immediate questions: what permissions does this key have, explicitly and through policy inheritance? What other resources has it touched in the past 30 days? Has it been used from IP addresses or user agents inconsistent with its normal pattern? Has it been used to create or modify other credentials?

That last question matters more than most IR teams account for. Attackers with access to IAM credentials routinely create new keys, add permissions to existing roles, or establish cross-account trusts to preserve access beyond the rotation of the original credential. Rotating one key while missing the three others created during the dwell period produces the same situation described in every post-eradication reinvestigation: you evicted the attacker from the front door and left all the others open.

Containment Without a Cable to Pull

Traditional containment isolates a compromised host by removing it from the network. Cloud containment looks different.

Isolate an EC2 instance by modifying its security group to deny all inbound and outbound traffic except from your forensics workstation. Take a snapshot of the EBS volumes before anything else, because snapshots are your disk image equivalent, and the instance may be terminated by an autoscaling policy at any point. If you need memory forensics, you need to act before the instance is terminated or rebooted.

For IAM keys, containment means immediate rotation and attaching a deny-all policy to the compromised principal, not just disabling the key. Disabled keys can be re-enabled. A deny policy attached to the user or role is more robust. Revoke all active sessions associated with the credential using the relevant API calls before assuming the key is contained.

AWS, Azure, and GCP each have mechanisms to isolate resources at the account, subscription, and project level respectively. These are the options worth knowing before you need them, because figuring out how to quarantine a production AWS account during a crisis is a bad learning environment.

What Eradication Actually Involves

Eradication in cloud means reaching a state where you can account for every access path the attacker had. That is a higher bar than it sounds.

Audit all IAM users, roles, access keys, and external identity federation configurations for anything created or modified during the incident window. Check for new cross-account trust relationships, unexpected service control policy changes, or newly created SAML identity providers. Verify that CloudTrail itself was not modified or disabled during the dwell period, because disabling audit logging is among the first things a knowledgeable cloud attacker does.

Check every region. Cloud resources exist in every region, and incident scope usually starts with the region where the alert fired. Attackers who know cloud environments use regions outside the ones being monitored. IAM is global, but EC2 instances, Lambda functions, and storage resources are regional. A complete eradication review covers all regions, including the ones you would not normally look at.

The Preparation That Makes This Possible

Cloud IR without preparation is archaeology. You are piecing together what happened from fragments of logs, hoping the relevant data event logging was enabled, hoping the trail was not tampered with, hoping the relevant instances have not already been terminated and replaced by autoscaling.

The organizations that handle cloud incidents well run IR readiness assessments before incidents happen. They verify logging configuration, test their ability to query CloudTrail across accounts and regions at scale, confirm that forensic snapshot procedures work in their environment, and define cloud-specific containment runbooks for their common resource types. They know which teams own which AWS accounts before the phone rings at 2 AM.

The traditional playbook handles on-premises incidents well. It was built for them. Cloud incidents follow a different logic, and the gap between the two playbooks is exactly the space a motivated attacker will use. The first step toward closing that gap is acknowledging the playbook does not currently cover it.