Labsco
DataDog logo

dd-monitors

★ 940

by datadog-labs · part of DataDog/pup

Monitor management - create, update, mute, and alerting best practices.

🔥🔥🔥✓ VerifiedFreeQuick setup
#automation#slack#git#monitoring
🧰 Not standalone. This skill ships with DataDog/pup and only works together with that tool — install the tool first, then add this skill.

Monitor management - create, update, mute, and alerting best practices.

Inspect the full instructions your agent will receiveExpand

This is the exact playbook injected into your agent when the skill activates — shown here so you can audit it before installing. You don't need to read it to use the skill.

by datadog-labs

Monitor management - create, update, mute, and alerting best practices. npx skills add https://github.com/datadog-labs/pup --skill dd-monitors Download ZIPGitHub940

Datadog Monitors

Create, manage, and maintain monitors for alerting.

Common Operations

List Monitors

Copy & paste — that's it
pup monitors list
pup monitors list --tags "team:platform"
pup monitors search --query "status:Alert"

Get Monitor

Copy & paste — that's it
pup monitors get

Create Monitor

Copy & paste — that's it
pup monitors create --file monitor.json

Mute/Unmute

Copy & paste — that's it
# Mute with duration
pup monitors update 12345 --file monitor-muted.json

# Or mute with specific end time
pup monitors update 12345 --file monitor-muted-until.json

# Unmute
pup monitors update 12345 --file monitor-unmuted.json

⚠️ Monitor Creation Best Practices

1. Avoid Alert Fatigue

Rule Why No flapping alerts Use last_Xm not last_1m Meaningful thresholds Based on SLOs, not guesses Actionable alerts If no action needed, don't alert Include runbook @runbook-url in message

Copy & paste — that's it
# WRONG - will flap constantly
query = "avg(last_1m):avg:system.cpu.user{*} > 50" # ❌ Too sensitive

# CORRECT - stable alerting
query = "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80" # ✅ Reasonable window

2. Use Proper Scoping

Copy & paste — that's it
# WRONG - alerts on everything
query = "avg(last_5m):avg:system.cpu.user{*} > 80" # ❌ No scope

# CORRECT - scoped to what matters
query = "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80" # ✅

3. Set Recovery Thresholds

Copy & paste — that's it
monitor = {
 "query": "avg(last_5m):avg:system.cpu.user{env:prod} > 80",
 "options": {
 "thresholds": {
 "critical": 80,
 "critical_recovery": 70, # ✅ Prevents flapping
 "warning": 60,
 "warning_recovery": 50
 }
 }
}

4. Include Context in Messages

Copy & paste — that's it
message = """

## High CPU Alert

Host: {{host.name}}
Current Value: {{value}}
Threshold: {{threshold}}

### Runbook
1. Check top processes: `ssh {{host.name}} 'top -bn1 | head -20'`
2. Check recent deploys
3. Scale if needed

@slack-ops @pagerduty-oncall
"""

⚠️ NEVER Delete Monitors Directly

Use safe deletion workflow (same as dashboards):

Copy & paste — that's it
def safe_mark_monitor_for_deletion(monitor_id: str, client) -> bool:
 """Mark monitor instead of deleting."""
 monitor = client.get_monitor(monitor_id)
 name = monitor.get("name", "")
 
 if "[MARKED FOR DELETION]" in name:
 print(f"Already marked: {name}")
 return False
 
 new_name = f"[MARKED FOR DELETION] {name}"
 client.update_monitor(monitor_id, {"name": new_name})
 print(f"✓ Marked: {new_name}")
 return True

Monitor Types

Type Use Case metric alert CPU, memory, custom metrics query alert Complex metric queries service check Agent check status event alert Event stream patterns log alert Log pattern matching composite Combine multiple monitors apm APM metrics

Audit Monitors

Copy & paste — that's it
# Find monitors without owners
pup monitors list | jq '.[] | select(.tags | contains(["team:"]) | not) | {id, name}'

# Find noisy monitors (high alert count)
pup monitors list | jq 'sort_by(.overall_state_modified) | .[:10] | .[] | {id, name, status: .overall_state}'

Downtime vs Muting

Use When Mute monitor Quick one-off, < 1 hour Downtime Scheduled maintenance, recurring

Copy & paste — that's it
# Downtime (preferred)
pup downtime create --file downtime.json

Failure Handling

Problem Fix Alert not firing Check query returns data, thresholds Too many alerts Increase window, add recovery threshold No data alerts Check agent connectivity, metric exists Auth error pup auth refresh

References