Shell & Log Scripting
The Unix philosophy: small programs, each doing one thing, joined by a pipe. The pipeline awk | sort | uniq -c extracts a field, groups it, and counts it in one line — the thing that would take a loop in another language. For an on-call engineer it's the first incident tool: the log is on disk, the terminal is right there, the answer is seconds away.
The topic covers both theory and practice. The main trap: uniq collapses only adjacent duplicates, so without a preceding sort it is silently wrong. The second: confusion over awk field numbering ($1, $2, a custom -F), making the pipeline emit the wrong column. The third is algorithmic: when the log is huge, a full sort is overkill, and the top-K is taken with a bounded heap at O(N log K). The breakdown is in the layers below.
Topic map
- Shell pipelines — how
|joins commands as a stream and whatawk,sort,uniqeach contribute. - Log analysis — counting unique values and frequencies by a log field with
awk | sort | uniq. - Reading pipeline output — predicting what a multi-stage
awkprints from field numbering and separators. - Top-K frequency — finding the K most frequent items at
O(N log K)with a bounded heap instead of a full sort.
Common traps
| Mistake | Consequence |
|---|---|
Using uniq without a preceding sort | Non-adjacent duplicates survive — the count is wrong |
| Counting raw lines instead of distinct field values | wc -l gives the line count, not the unique-value count |
Extracting the wrong field in awk | The wrong column is silently counted ($1 vs $2) |
Thinking -F: keeps the part after the separator | awk -F: '{print $1}' takes the part before the first separator |
| Forgetting a pipeline is streaming | Commands run concurrently, not via a temp file |
| Fully sorting all uniques for top-K | O(U log U) where O(N log K) with a heap suffices |
| Using a max-heap for top-K instead of a min-heap | The global max is evicted, not the needed smallest of the top-K |
Interview relevance
These tasks test whether you think in pipelines and understand what each tool does. A candidate who writes awk '{print $1}' log | sort -u | wc -l on the spot and explains why sort is required before uniq shows working fluency, not a memorized recipe.
Typical checks:
- The roles of
awk,sort,uniqindividually and whyuniqrequires a sort. - Working a specific pipeline by field numbering — what ends up on screen.
- Estimating complexity: where the bottleneck is — the sort at
O(n log n). - The algorithmic jump for top-K — a bounded heap instead of a full sort.
Common wrong answer: "uniq will remove all duplicates by itself". No — it collapses only adjacent repeats, so without sort the result is wrong. And "for top-K I'll sort the whole file" — with K ≪ U that's wasted O(U log U) work instead of O(N log K).