Labsco
datadog-labs logo

dd-audit-key-compromise

★ 139

by datadog-labs · part of datadog-labs/agent-skills

Investigate a potentially compromised Datadog API key — timeline of actions, geo/IP breakdown, endpoints called, anomaly flags, and remediation steps.

🔥🔥🔥✓ VerifiedFreeQuick setup
#git#monitoring
🧩 One of 7 skills in the datadog-labs/agent-skills package — works on its own, and pairs well with its siblings.

This is the playbook your agent receives when the skill activates — you don't need to read it to use the skill, but it's here to audit before installing.

Audit Trail: API Key Compromise Investigation

Reconstruct what a Datadog API key did, where requests originated, and which resources were affected.

Investigation Workflow

Step 1 — Establish timeline

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      endpoint: .attributes.attributes.http.url_details.path,
      method: .attributes.attributes.http.method,
      ip: .attributes.attributes.network.client.ip,
      city: .attributes.attributes.network.client.geoip.city.name,
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name
    }]'

Step 2 — Geo/IP breakdown

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
  | jq '[.data[] | {
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name,
      ip: .attributes.attributes.network.client.ip
    }]
    | group_by(.country)
    | map({
        country: .[0].country,
        count: length,
        asns: [.[].asn] | unique,
        ips: [.[].ip] | unique
      })
    | sort_by(-.count)'

Step 3 — Endpoint breakdown

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
  | jq '[.data[] | {
      method: .attributes.attributes.http.method,
      path: .attributes.attributes.http.url_details.path
    }]
    | group_by(.path)
    | map({path: .[0].path, methods: [.[].method] | unique, count: length})
    | sort_by(-.count)'

Step 4 — Destructive action check

pup audit-logs search --query "@metadata.api_key.id:KEY_ID @action:deleted" --from 90d -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Step 5 — When was the key created and by whom?

pup audit-logs search --query "@asset.type:api_key @asset.id:KEY_ID @action:created" --from 90d -o json \
  | jq '[.data[] | {
      created_at: .attributes.timestamp,
      created_by: .attributes.attributes.usr.email,
      creator_ip: .attributes.attributes.network.client.ip,
      creator_country: .attributes.attributes.network.client.geoip.country.name
    }]'

Anomaly Flags

SignalWhy it matters
Country not in org's normal baselinePossible exfiltration from unexpected region
ASN is a cloud/VPN provider (AWS, Cloudflare, NordVPN, etc.)Proxied traffic; obscured origin
DELETE actions on monitors, dashboards, or log pipelinesPotential sabotage
Burst of activity in short windowAutomated scraping or bulk exfiltration
Activity outside business hoursOff-hours access
Key used from multiple IPs simultaneouslyKey shared or stolen

Investigation Output Format

Key ID: <key_id>
Created: <timestamp> by <user_email>
Active period: <first_seen> to <last_seen>
Total events: <N>

Origins:
  - <Country> (<ASN>): <N> events — [NORMAL / FLAG: first-time origin]

Endpoints called (top 5):
  - <METHOD> <path>: <N> calls

Destructive actions: <N> deletions — [resource types affected]

Recommended actions:
  1. Revoke the key immediately if not already done
  2. Review affected resources: [list]
  3. Check if any deleted resources need restoration
  4. Audit who else had access to this key

Remediation

Revoke in Datadog UI: Organization Settings > API Keys > Revoke.

Or via API (requires manage_api_keys scope):

pup api-keys delete KEY_ID

References