Observability & Monitoring
Monitoring answers "is the service healthy right now?", while observability answers "why is it behaving this way?". The system continuously collects signals about hosts and services, stores them over time, visualizes them, and raises alerts when a threshold is crossed. The goal is to detect failures and degradation early and to provide data for diagnosis before a problem becomes an incident.
The topic ties together four senior ops skills. First, understanding why monitoring exists and what tools build it. Second, distinguishing the three pillars: metrics answer "is it healthy?", logs "what happened?", traces "where is it slow?". Third, reading specific signals, above all load average, and localizing the bottleneck from them (CPU, disk, memory). Fourth, bringing hundreds of hosts to a single state idempotently, not by hand. The breakdown is in the layers below.
Topic map
- Monitoring fundamentals — the purpose of a monitoring system, the "collect → store → visualize → alert" flow, and the tools.
- Metrics, logs and traces — the three pillars of observability and the question each answers.
- Load average — what load average measures, how to read its trend, and its relation to the core count.
- Configuration management — idempotent tools like Ansible and rolling out agents to thousands of hosts.
Common traps
| Mistake | Consequence |
|---|---|
| Treating metrics, logs, and traces as interchangeable | The wrong tool for the question — diagnosis stalls |
| Reading load average without the core count | "50 is a disaster" on a 64-core host is a false alarm |
| Confusing high load with high CPU | Load can rise from I/O wait while the CPU is idle |
| Keeping only logs, no metrics | No cheap aggregates or trends — the failure shows late |
| Configuring hosts by hand | Configuration drift; unmanageable at 1000 hosts |
| Thinking a playbook is not idempotent | Fear of re-running; the tool's key property is lost |
Interview relevance
At the senior level observability is tested through practice: they show a top screen or a set of metrics and ask you to name the bottleneck. A candidate who, from "load 38 on 4 cores, CPU idle, wa 86%", immediately says "the host is disk-bound, not CPU-bound" demonstrates exactly what the question is for — diagnosing from signals.
Typical checks:
- The purpose of monitoring and the typical tools (Prometheus, Zabbix, Grafana, ELK).
- The three pillars and the question each answers.
- Reading load average relative to the core count and by the 1/5/15-minute trend.
- Idempotency and rolling out an agent to thousands of hosts via configuration management.
Common wrong answer: "a high load average means the CPU is overloaded". Not necessarily: load also counts processes in uninterruptible I/O wait, so with high wa and an idle CPU the disk is to blame. And "I'll configure hosts by hand over SSH" — at scale this leads to configuration drift.