runbook-creation

Imported from sickn33/agentic-awesome-skills (skills/runbook-creation/SKILL.md). Install upstream with npx skills add sickn33/agentic-awesome-skills --skill runbook-creation. Copyright stays with the author (MIT).

Runbook Creation

Create effective operational runbooks, standard operating procedures, and troubleshooting guides that any on-call engineer can follow under pressure.

Runbook Template — Full Structure

# Runbook: [Service / Process Name]

**Owner:** [Team or individual]
**Last Reviewed:** YYYY-MM-DD
**Version:** X.Y
**Severity if unavailable:** SEV[1-4]

---

## Overview

Brief description of the service, why this runbook exists, and when to
use it.

## Prerequisites

- [ ] Required access / IAM role: [details]
- [ ] Tools installed: [kubectl, aws-cli, psql, etc.]
- [ ] VPN connected to [environment]
- [ ] Communication channel open: [Slack #channel]

## Procedure

### Step 1 — [Action Name]

[Explanation of what this step does and why.]

```bash
# command here
```

**Expected output:** [describe what success looks like]

### Step 2 — [Action Name]

```bash
# command here
```

**Expected output:** [description]

*(Continue with numbered steps...)*

## Verification

How to confirm the procedure succeeded:

- [ ] [Check 1 — e.g., health endpoint returns 200]
- [ ] [Check 2 — e.g., no errors in logs for 5 minutes]
- [ ] [Check 3 — e.g., metrics return to baseline]

## Rollback

If the procedure fails or causes unexpected issues:

### Rollback Step 1
```bash
# rollback command
```

### Rollback Step 2
```bash
# rollback command
```

## Troubleshooting

| Symptom | Likely Cause | Resolution |
|---------|-------------|------------|
| [symptom 1] | [cause] | [fix] |
| [symptom 2] | [cause] | [fix] |

## Escalation

If unresolved after [X] minutes:
- **Primary:** @[team-lead] — [phone/Slack]
- **Secondary:** @[manager] — [phone/Slack]

## Related Runbooks

- [Link to related runbook 1]
- [Link to related runbook 2]

## Change Log

| Date | Author | Change |
|------|--------|--------|
| YYYY-MM-DD | [Name] | Initial version |

Overview

Failover the primary PostgreSQL instance to the synchronous replica when the primary is unreachable or degraded. This runbook covers both planned (maintenance) and unplanned (emergency) failover.

Prerequisites

• DBA or SRE-level access to primary and replica hosts
• psql client installed (v14+)
• VPN connected to production network
• Slack channel #db-ops open
• Confirm replica is in sync: replication lag < 1 MB

Procedure

Step 1 — Verify Replica Health

psql -h replica.db.internal -U dba -d postgres -c \
  "SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn();"

Expected output: pg_is_in_recovery = t, LSN advancing.

Step 2 — Stop Application Writes

kubectl scale deployment api-server --replicas=0 -n production
kubectl scale deployment worker --replicas=0 -n production

Expected output: Deployments scaled to 0 pods.

Step 3 — Fence the Old Primary and Confirm Write Quiescence

Before promotion, use the documented failover manager/fencing mechanism to ensure the old primary cannot accept writes, including after network recovery. Scaling two deployments and observing zero active queries is not proof of fencing or absence of other writers. If the primary is unreachable and fencing cannot be confirmed, stop and escalate; do not risk two primaries. Capture original replica counts/configuration for restoration.

psql -h primary.db.internal -U dba -d postgres -c \
  "SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND query !~ 'pg_stat';"

Expected output: Count = 0 (no active queries).

Step 4 — Promote Replica

psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_promote();"

Wait up to 30 seconds, then confirm:

psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_is_in_recovery();"

Expected output: pg_is_in_recovery = f (no longer a replica).

Step 5 — Update DNS

aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "db.internal.example.com",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [{"Value": "replica.db.internal"}]
      }
    }]
  }'

Step 6 — Restart Application

kubectl scale deployment api-server --replicas=6 -n production
kubectl scale deployment worker --replicas=4 -n production

Verification

• psql -h db.internal.example.com -c "SELECT 1;" returns successfully
• Application logs show successful DB connections (no errors for 5 min)
• Transaction throughput returns to baseline on Grafana dashboard
• No replication-lag alerts firing

Rollback

Promotion is not undone by restoring a backup. Establish the authoritative primary, fence other writers and obtain a DBA-approved data-recovery/rejoin plan. The following restores a separate RDS instance and may lose writes since the snapshot; it is not an automatic rollback for the self-managed PostgreSQL example:

# Restore latest automated snapshot (RDS example)
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier prod-db-restored \
  --db-snapshot-identifier prod-db-latest-snapshot

Escalation

If unresolved after 15 minutes:

• Primary: @dba-lead — +1-555-0101
• Secondary: @platform-oncall — +1-555-0102

The shell fragments assume Linux/GNU date, curl, jq/openssl and authorized cluster/cloud access. Use bounded timeouts for network probes and protect collected logs/queries; a fresh temporary directory does not make raw incident data safe to publish.

## Automation Scripts for Common Operations

### Service Health Check

```bash
#!/usr/bin/env bash
# health-check.sh — Check health of critical services
set -euo pipefail

SERVICES=(
  "https://api.example.com/healthz"
  "https://app.example.com/healthz"
  "https://admin.example.com/healthz"
)

EXIT_CODE=0

for url in "${SERVICES[@]}"; do
  if ! HTTP_CODE=$(curl -so /dev/null -w '%{http_code}' --max-time 5 "$url" 2>/dev/null); then
    HTTP_CODE=000
  fi
  if [ "$HTTP_CODE" -eq 200 ]; then
    printf "  OK    %s\n" "$url"
  else
    printf "  FAIL  %s (HTTP %s)\n" "$url" "$HTTP_CODE"
    EXIT_CODE=1
  fi
done

exit $EXIT_CODE
```

### Log Collection for Incident Investigation

```bash
#!/usr/bin/env bash
# collect-logs.sh — Gather logs from multiple sources for incident review
set -euo pipefail

INCIDENT_ID="${1:?Usage: collect-logs.sh <incident-id>}"
[[ "$INCIDENT_ID" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2
umask 077
OUTDIR=$(mktemp -d "${TMPDIR:-/tmp}/incident-${INCIDENT_ID}.XXXXXX")

echo "Collecting logs for incident $INCIDENT_ID..."

# Kubernetes pod logs (last 30 min)
kubectl logs -l app=api-server -n production --since=30m \
  > "${OUTDIR}/api-server-pods.log" 2>&1

# CloudWatch Logs (last 30 min)
aws logs filter-log-events \
  --log-group-name /ecs/production/api \
  --start-time "$(date -d '30 minutes ago' +%s)000" \
  --output text > "${OUTDIR}/cloudwatch-api.log" 2>&1

# Database slow query log
psql -h db.internal -U dba -d postgres -c \
  "SELECT * FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start;" \
  > "${OUTDIR}/db-active-queries.log" 2>&1

# System resource snapshot
kubectl top pods -n production > "${OUTDIR}/pod-resources.log" 2>&1

echo "Logs saved to $OUTDIR"
tar czf "${OUTDIR}.tar.gz" -C "$(dirname "$OUTDIR")" "$(basename "$OUTDIR")"
echo "Archive: ${OUTDIR}.tar.gz"
```

### Certificate Expiry Check

```bash
#!/usr/bin/env bash
# cert-check.sh — Warn if TLS certificates expire within 30 days
set -euo pipefail

DOMAINS=(
  "api.example.com"
  "app.example.com"
  "admin.example.com"
)

WARN_DAYS=30
TODAY=$(date +%s)
EXIT_CODE=0

for domain in "${DOMAINS[@]}"; do
  if ! EXPIRY=$(echo | openssl s_client -servername "$domain" -connect "${domain}:443" 2>/dev/null \
    | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2); then
    printf "  FAIL  %s certificate unavailable\n" "$domain"
    EXIT_CODE=1
    continue
  fi
  EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null || echo 0)
  DAYS_LEFT=$(( (EXPIRY_EPOCH - TODAY) / 86400 ))

  if [ "$DAYS_LEFT" -lt "$WARN_DAYS" ]; then
    printf "  WARN  %s expires in %d days (%s)\n" "$domain" "$DAYS_LEFT" "$EXPIRY"
    EXIT_CODE=1
  else
    printf "  OK    %s — %d days remaining\n" "$domain" "$DAYS_LEFT"
  fi
done

exit $EXIT_CODE
```

### Disk Space Cleanup

```bash
#!/usr/bin/env bash
# disk-cleanup.sh — Free disk space on a host
set -euo pipefail

echo "=== Disk Usage Before ==="
df -h /

# Inventory first; no deletion in this diagnostic example.
journalctl --disk-usage
docker system df
find /var/log -type f -name "*.gz" -mtime +7 -print
# Use service-specific approved retention and reviewed exact paths for cleanup.
# Do not prune volumes or globally delete /tmp; backups/log evidence may be needed.

echo "=== Disk Usage After ==="
df -h /
```

## Runbook Review Checklist

Use this checklist every time a runbook is created or updated.

```yaml
content_review:
  - [ ] Title clearly identifies the service and operation
  - [ ] Overview explains WHEN and WHY to use this runbook
  - [ ] Prerequisites list all required access, tools, and setup
  - [ ] Every step has a concrete command (no vague instructions)
  - [ ] Expected output is documented for each step
  - [ ] Verification section confirms success with specific checks
  - [ ] Rollback section exists and has been tested
  - [ ] Escalation contacts are current (names, phones, Slack handles)
  - [ ] Troubleshooting table covers the top 3-5 known failure modes

usability_review:
  - [ ] A new team member can follow the runbook without tribal knowledge
  - [ ] Steps are numbered and sequential (no branching without clear labels)
  - [ ] Commands can be copy-pasted (no placeholder values without explanation)
  - [ ] Time estimates included for long-running steps
  - [ ] No jargon or acronyms used without definition

maintenance_review:
  - [ ] Owner and last-reviewed date are set
  - [ ] Version number incremented
  - [ ] Change log entry added
  - [ ] Related runbooks section is up to date
  - [ ] Links to dashboards and docs are valid (not broken)
```

## Runbook Testing Procedures

```yaml
testing_strategy:
  dry_run:
    frequency: "Every time a runbook is created or substantially edited"
    method: "Walk through each step in a staging environment"
    goal: "Verify commands work and output matches documentation"

  peer_review:
    frequency: "Every edit"
    method: "Another engineer follows the runbook in staging without help"
    goal: "Confirm the runbook is self-contained and unambiguous"

  scheduled_validation:
    frequency: "Quarterly"
    method: "SRE team picks 5 runbooks at random, executes in staging"
    goal: "Catch runbooks that have drifted from production reality"

  incident_triggered:
    trigger: "Any time a runbook is used in a real incident"
    method: "Post-mortem includes runbook accuracy assessment"
    goal: "Capture improvements while the experience is fresh"

  automation_testing:
    method: "CI pipeline validates bash scripts with shellcheck and dry-run"
    example: |
      # .github/workflows/runbook-lint.yml
      name: Lint Runbook Scripts
      on: [pull_request]
      jobs:
        shellcheck:
          runs-on: ubuntu-latest
          steps:
            - uses: actions/checkout@v4
            - name: ShellCheck
              run: |
                find runbooks/ -name "*.sh" -exec shellcheck {} +
```



## When to Use

- Creating or updating operational runbooks, SOPs, on-call procedures, or incident recovery guides.

## Limitations

- Runbooks must be validated in staging; examples use placeholder hostnames and account IDs.
- Docs-only import: upstream template files are not bundled.

### Example

```markdown
Draft a SEV2 database failover runbook with prerequisites, verification, and rollback.
```

> Adapted from [BagelHole/DevOps-Security-Agent-Skills](https://github.com/BagelHole/DevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.