Amazon CloudWatch¶

Three pillars of observability:
  Metrics  → numbers over time (CPU = 78%, RequestCount = 4,320)
  Logs     → text records of events (application logs, access logs, Lambda output)
  Traces   → request path across distributed services (X-Ray integration)

CloudWatch covers: Metrics + Logs
AWS X-Ray covers:  Traces (distributed tracing)

Metric anatomy:
  Namespace:  AWS/EC2          ← service grouping (or custom: MyApp/Orders)
  MetricName: CPUUtilization   ← what is measured
  Dimensions: InstanceId=i-1234567890abcdef0  ← what it applies to
  Unit:       Percent
  Value:      78.3
  Timestamp:  2026-04-08T15:30:00Z

AWS/EC2 namespace:
  CPUUtilization         ← % CPU used
  NetworkIn / NetworkOut ← bytes transferred
  DiskReadOps / DiskWriteOps ← disk operations
  StatusCheckFailed      ← instance or system check failure

NOT available by default (need CloudWatch Agent):
  Memory (RAM) utilization  ← OS-level, AWS cannot see it
  Disk space utilization    ← OS-level, AWS cannot see it
  Process counts            ← OS-level

CloudWatch Agent:
  Installed on EC2 → reads OS-level metrics → sends to CloudWatch
  IAM role required: CloudWatchAgentServerPolicy
  Config: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Standard resolution: 1 minute  (minimum for detailed monitoring)
High resolution:     1 second  (custom metrics only — higher cost)

Retention:
  < 60 seconds resolution: retained 3 hours
  1-minute resolution:     retained 15 days
  5-minute resolution:     retained 63 days
  1-hour resolution:       retained 455 days (15 months)
  (CloudWatch automatically aggregates older data to lower resolution)

Example: Error rate % from two metrics
  m1: HTTPCode_ELB_5XX_Count
  m2: RequestCount
  Expression: (m1/m2)*100  → ErrorRate %
  → Create alarm on ErrorRate instead of raw counts

Functions available:
  METRICS()      → array of all metrics in expression
  SUM(METRICS()) → sum across all dimensions (e.g., total CPU across fleet)
  AVG(METRICS()) → average across all dimensions
  MIN/MAX        → minimum/maximum of metrics
  ANOMALY_DETECTION_BAND(m1, 2) → ML band for anomaly detection [docs.aws.amazon](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html)

OK          → metric within acceptable range
ALARM       → metric exceeded threshold → actions triggered
INSUFFICIENT_DATA → not enough data points yet (new alarm or metric gap)

Metric:          what to watch (CPUUtilization of i-1234567890abcdef0)
Threshold:       > 80%
Period:          300 seconds (5 minutes) — data point interval
Evaluation:      3 out of 3 periods → must breach 3 consecutive periods
Datapoints:      N of M evaluation (e.g., 3 out of 5 periods)

# Alarm → SNS → Lambda → auto-remediate
Alarm: EC2 CPU > 90% for 3 periods
  → SNS topic: ops-alerts
    → Email: on-call engineer
    → Lambda: snapshot instance + send PagerDuty alert

Multiple alarms combined with AND/OR logic:
  ALARM if: (CPU > 90% AND Memory > 85%)   ← avoid false positives
  ALARM if: (5xx errors > 100 OR latency > 5s)

Benefits:
  Reduce alert noise — only alarm when multiple signals agree
  Create a "service health" rollup alarm from individual metric alarms

Instead of static threshold → use ML-learned band of expected values:

CloudWatch trains model on 2 weeks of metric history
  → Learns hourly, daily, weekly patterns
  → Creates expected value band (configurable width in std deviations)

Alarm triggers when metric goes OUTSIDE the band:
  ANOMALY_DETECTION_BAND(m1, 2)  ← 2 = number of standard deviations

Example:
  Normal traffic: 1,000–5,000 req/min (varies by time of day)
  Static threshold alarm: > 8,000 → many false positives on peak hours
  Anomaly detection: > expected range for this time of day → accurate

Use cases:
  Traffic spikes that are unusual for the current time
  Lambda duration deviating from normal
  Error rates deviating from baseline
  Business metrics (orders/minute) dropping unexpectedly

Log Group   → container for a service/application
  └── Log Stream → sequence of events from one source (one EC2, one Lambda)
       └── Log Events → individual timestamped log entries

Default: logs kept forever (never expire) — accrues cost indefinitely
Set retention: 1 day, 3 days, 5 days, 1 week, 2 weeks, 1/3/6 months, 1/2/5/10 years

Best practice:
  Dev log groups:  7 days
  Prod app logs:   90 days → then archive to S3 via subscription
  Audit logs:      1–7 years (compliance)

Log group: /aws/lambda/order-processor
Filter pattern: [timestamp, requestId, level="ERROR", ...]
→ Creates metric: LambdaErrors (count of matching lines)
→ Create alarm on this metric

Example patterns:
  "ERROR"                          ← any line containing ERROR
  "[ERROR]"                        ← literal [ERROR]
  { $.statusCode = 500 }           ← JSON log: statusCode field = 500
  { $.latency > 1000 }             ← JSON log: latency > 1000ms

-- Most common query pattern: find errors in last 1 hour
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

-- Top 10 slowest Lambda invocations
fields @timestamp, @duration, @requestId
| filter @type = "REPORT"
| sort @duration desc
| limit 10

-- Error rate percentage
fields @timestamp, @message
| stats
    count(*) as totalRequests,
    sum(@message like /ERROR/) as errors
| project errors/totalRequests * 100 as errorRate

-- Parse custom log format
fields @message
| parse @message "user=* action=* duration=*ms" as user, action, duration
| stats avg(duration) by action
| sort avg_duration desc

-- Lambda cold starts
fields @timestamp, @initDuration
| filter @initDuration > 0
| stats count() as coldStarts, avg(@initDuration) as avgInitMs

-- VPC Flow Log: top talkers
fields srcAddr, dstAddr, bytes
| stats sum(bytes) as totalBytes by srcAddr, dstAddr
| sort totalBytes desc
| limit 10

Key commands:
  fields    → select specific fields to return
  filter    → where clause (like SQL WHERE)
  stats     → aggregate: count, sum, avg, min, max, percentile
  sort      → order results
  limit     → cap result count
  parse     → extract fields from unstructured log text
  dedup     → remove duplicates by field
  display   → rename/format fields in output

Time range: query logs from any time window (up to retention period)
Supports: all log groups simultaneously (cross-log-group query)

Stream logs in real-time to:
  Lambda           → process and forward to Elasticsearch/Splunk/custom
  Kinesis Data Streams    → high-volume real-time processing
  Kinesis Firehose → buffer and deliver to S3/Datadog/Splunk

Use case:
  /aws/lambda/api → subscription filter → Kinesis → Firehose → S3
  → Centralized log archive at low cost

Export historical log data to S3:
  CreateExportTask API → exports to S3 (takes minutes to hours)
  Not real-time — up to 12 hours delay

Use for: compliance archiving, bulk analysis in Athena

Collects:
  Memory utilization (RAM)
  Disk space per mount point
  Disk I/O per device
  Network metrics per interface
  Process metrics
  Custom app logs (tail any log file)
  StatsD / collectd metrics from applications

Install:
  SSM Run Command (recommended for fleets)
  Or manually: yum install amazon-cloudwatch-agent

Configure:
  /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
  Or use: aws-cloudwatch-agent-config-wizard (interactive)

IAM role needed:
  CloudWatchAgentServerPolicy (attach to EC2 instance profile)

Global service — view metrics from any region on one dashboard
Share with: AWS accounts, email (public link), third parties (no AWS login)

Widget types:
  Line chart     → trends over time
  Number         → current metric value (CPUUtilization: 34%)
  Gauge          → visual meter
  Bar chart      → comparison
  Text           → markdown annotations and runbook links
  Alarm status   → traffic light for multiple alarms
  Log table      → embedded Logs Insights query result
  Explorer       → auto-discover and graph tagged resources

Automatic dashboards:
  AWS creates default dashboards for each service → CloudWatch → Dashboards → Automatic
  EC2, Lambda, RDS, ELB etc. — pre-built, no configuration needed

Runs scripted synthetic transactions against your application endpoints
  → Detects issues BEFORE real users do
  → Monitors availability and performance from outside your system

Canary types:
  Heartbeat monitor:  GET your URL → check 200 response
  API canary:         series of API calls → validate responses
  Broken link check:  crawl pages → find broken links
  GUI workflow:       Puppeteer script → simulate user login/checkout
  Visual monitoring:  screenshot comparison (detects UI regressions)

Schedule: every 1 minute to every 1 hour
Stores: screenshots, HAR files, Lambda execution logs in S3
Metrics: SuccessPercent, Duration → create alarms on them

Use case:
  Alarm: canary SuccessPercent < 100% for 3 minutes
  → Trigger PagerDuty before your monitoring team notices

ServiceLens = CloudWatch Metrics + Logs + X-Ray Traces unified view
  → Map of microservices with health, latency, error rate per service
  → Click any service → drill into its traces
  → Click any trace → see which Lambda/EC2/DynamoDB call is slow

X-Ray adds to CloudWatch:
  Distributed tracing: follow one request across Lambda → API Gateway → DynamoDB
  Service map: visual dependency graph with latency/error annotations
  Segments + subsegments: timing breakdown of each component

Enable tracing:
  Lambda: X-Ray active tracing → one checkbox
  EC2: install X-Ray daemon
  ECS: X-Ray sidecar container

Metrics:
  AWS default metrics:  free
  Custom metrics:       $0.30/metric/month (first 10,000 metrics)
  High-resolution:      $0.02/metric/month additional

Alarms:
  Standard resolution:  $0.10/alarm/month
  High-resolution:      $0.30/alarm/month
  Composite alarms:     $0.50/alarm/month

Logs:
  Ingestion:    $0.50/GB
  Storage:      $0.03/GB/month
  Insights query: $0.005/GB scanned

Dashboards:
  First 3 dashboards: free
  After: $3/dashboard/month

Synthetics canaries:
  $0.0012 per canary run