Observability & Monitoring

A monitoring system continuously collects signals about hosts and services, stores them over time, visualizes them, and raises alerts when something crosses a threshold. Its purpose is to detect failures and degradation early and give data to diagnose them. Common tools include Prometheus, Zabbix, Grafana, and the ELK stack.

Metrics are numeric time series — cheap aggregates like request rate or CPU showing a service's state and trends. Logs are timestamped discrete events with detail, used to find what happened around an error. Traces follow one request across services, exposing latency to locate bottlenecks. Metrics answer 'is it healthy?', logs 'what happened?', traces 'where is it slow?'

A counter only ever rises — requests or errors — and you read it with rate(). A gauge moves both ways, like memory or queue depth. A histogram buckets observations so you can derive latency percentiles from the buckets; a summary computes them client-side. Use a histogram for latency — an average hides the tail.

Load average is the average number of processes runnable or in uninterruptible wait over the last 1, 5, and 15 minutes. Here 50/20/10 means load is rising fast — recent load far exceeds the older window. Whether it is bad depends on core count: load near the number of cores is healthy, but far above it means tasks are queuing for CPU, disk, or I/O.

Error rate is rate() over the http_requests_total counter filtered to 5xx and summed over 5m. For p95 use histogram_quantile(0.95, ...) over the rate of the histogram _bucket series, but you must sum by (le) those buckets first. Never average latency, and never average percentiles across instances — aggregate buckets, then take the quantile.

Prometheus pulls — it scrapes targets on an interval via service discovery, so a missing scrape signals a target is down. Batch jobs too short-lived to scrape push to a Pushgateway that Prometheus scrapes. Alerting rules re-evaluate a PromQL expression each interval and fire once it holds for the for window — alert on symptoms, not causes.

Load average 38 on 4 cores is far above capacity, yet CPU is mostly idle and 86.5% is wa (I/O wait), and memory is fine — so the host is disk-bound, not CPU-bound. The bottleneck is disk I/O: processes are blocked waiting on storage. Confirm with iotop or iostat -x to find the busy device and offending process.

A configuration-management tool such as Ansible or Puppet declares the desired state of many hosts as code and brings each host to that state — packages, files, services. The tool makes only the changes needed. It is idempotent: re-running the same playbook on an already-correct host changes nothing, so runs are safe to repeat and converge to the declared state.

Do not do it by hand. Use configuration management (Ansible, Puppet, or Salt) to install the package, drop a templated config, and enable the service idempotently across all hosts — one re-runnable source of truth. Drive it in batches to avoid overloading the monitoring server, and use auto-registration gated by an allow-list so only expected hosts join. Expect: missing network access, OS/version heterogeneity, weak hosts, and credential distribution.

Flat CPU means the host is not the bottleneck. Apply RED — rate, errors, duration — per route: rising duration with a few errors points to a slow downstream dependency, not compute. USE — utilization, saturation, errors — stays low, ruling out CPU and memory. Read p99 from a histogram, never the mean, which hides the tail.

Scrape with Prometheus via service discovery so new Pods are found; store long-term in Thanos or Mimir; visualize in Grafana; alert via Alertmanager on SLO (service-level objective) burn, not raw errors. Add logs (Loki/ELK) and traces (Tempo/Jaeger) tied by a correlation id, so a dashboard shows whether the app or a dependency is at fault.