Shell & Log Scripting
Shell pipelines and log analysis with awk, sed, sort and uniq, plus algorithmic log-processing tasks.
14 questions
JuniorTheoryCommonWhat does set -euo pipefail do, flag by flag, and why use it?
What does set -euo pipefail do, flag by flag, and why use it?
It hardens a script's failure behaviour. -e exits on the first command that returns non-zero; -u treats an unset variable as an error instead of an empty string; -o pipefail makes a pipeline fail if ANY stage fails, not just the last. Without pipefail, cmd1 | cmd2 masks cmd1's failure whenever cmd2 succeeds.
Common mistakes
- ✗Believing
set -ealone catches a failure in the middle of a pipeline - ✗Expecting
-uto error on an already-set variable rather than an unset one - ✗Thinking
pipefailreports the last stage's status instead of the worst stage's
Follow-up questions
- →Why can
set -estill miss a failure insideif,&&, or a command substitution? - →How does adding
-o pipefailchange the exit code ofgrep x file | head?
MiddleCodeCommonAdd robust error handling and a cleanup trap to a fragile deploy script
Add robust error handling and a cleanup trap to a fragile deploy script
Turn on strict mode with set -euo pipefail so a failed command, a broken pipe stage, or an unset variable aborts the run. Right after mktemp, register trap 'rm -f "$tmp"' EXIT: the EXIT trap fires on a normal finish, an error abort, and a caught signal, so the temp file is always removed.
Common mistakes
- ✗Registering the
trapbeforemktemp, so an early failure never cleans up - ✗Believing plain
set -ealso removes temp files without atrap - ✗Using a
SIGINT- orRETURN-only trap, missing the error-abort exit path
Follow-up questions
- →Why register the
traponEXITrather than onSIGINTandSIGTERMseparately? - →How does
set -ebehave for a command whose failure you deliberately tolerate?
JuniorDebuggingOccasionalThis backup script silently archives the wrong paths — how do you debug and fix it?
This backup script silently archives the wrong paths — how do you debug and fix it?
Run it with bash -x script (or add set -x) and you see tar expanding to /srv/My and App as separate arguments — the unquoted $paths word-splits on every space. shellcheck flags it as SC2086. The fix is to store the paths in an array and pass "${paths[@]}", which keeps each path as one argument.
Common mistakes
- ✗Assuming multiple paths fit safely in one space-separated scalar variable
- ✗Not running
bash -xorshellcheck, so the silent word-split goes unseen - ✗Leaving
$pathsunquoted, soMy Appsplits into two arguments
Follow-up questions
- →What does
shellcheckcode SC2086 warn about, and how do you silence it correctly? - →Why does
set -xreveal a word-splitting bug that reading the source does not?
JuniorTheoryOccasionalIn a bash script, what are $0, $1, $@, and $#?
In a bash script, what are $0, $1, $@, and $#?
$0 is the script's own name or path; $1…$9 (and ${10} up) are the positional arguments; $# is the count of arguments; $@ is all of them. Always use "$@" in double quotes — it expands to each argument as a separate word, preserving arguments that contain spaces, whereas $* joins them into one string.
Common mistakes
- ✗Writing
$@unquoted, so arguments with spaces split into multiple words - ✗Confusing
$#(the argument count) with$0(the script name) - ✗Assuming
$*and"$@"behave the same when forwarding arguments
Follow-up questions
- →Why does
"$@"preserve an argument liketwo wordsbut$*does not? - →How do you
shiftpositional parameters to process them in a loop?
JuniorTheoryOccasionalHow do bash variables work — $var versus ${var}, command substitution, and quoting?
How do bash variables work — $var versus ${var}, command substitution, and quoting?
$var and ${var} both expand a variable; the braces only delimit the name, as in ${x}_bak. Command substitution $(cmd) captures a command's stdout into a value. The critical rule: an unquoted $var is word-split and glob-expanded, so a value with spaces becomes several arguments — always write "$var".
Common mistakes
- ✗Leaving
$varunquoted, so a value with spaces or globs word-splits into many arguments - ✗Thinking
${var}differs semantically from$varrather than just delimiting the name - ✗Confusing command substitution
$(cmd)with arithmetic expansion$((expr))
Follow-up questions
- →When is
${var}strictly required rather than merely optional? - →How does
"$@"differ from$*when forwarding arguments?
JuniorTheoryOccasionalWhat is an exit code, what does $? hold, and how do && and || chain on it?
What is an exit code, what does $? hold, and how do && and || chain on it?
Every command returns an exit code: 0 means success, any non-zero (1–255) means failure. $? holds the exit code of the most recent command. a && b runs b only if a succeeded (exit 0); a || b runs b only if a failed (non-zero). So mkdir d && cd d enters the directory only when the create worked.
Common mistakes
- ✗Reading
0as failure — in shells0is success and non-zero is failure - ✗Checking
$?after an intervening command, which has already overwritten it - ✗Swapping
&&and||—&&runs on success,||runs on failure
Follow-up questions
- →Why must you capture
$?immediately, before running any other command? - →How does
set -einteract with a command used in ana && bchain?
JuniorTheoryOccasionalWhat does a shell pipeline do, and what do awk, sort, and uniq each contribute?
What does a shell pipeline do, and what do awk, sort, and uniq each contribute?
A pipeline connects commands with |, feeding one command's stdout into the next's stdin so each does one job. awk extracts and transforms fields by column or pattern; sort orders lines, with -n numeric and -h human-readable; uniq collapses adjacent duplicate lines and -c counts them. The classic combo awk | sort | uniq -c extracts a field, groups it, and counts occurrences.
Common mistakes
- ✗Forgetting that
uniqonly collapses adjacent duplicates, so it needs a prior sort - ✗Confusing what
awk,sort, anduniqeach do - ✗Thinking a pipeline buffers to a temp file rather than streaming stdout to stdin
Follow-up questions
- →Why must you
sortbeforeuniq -cto get correct per-value counts? - →How would you add
sort -rnat the end to rank the counted values?
MiddleCodeOccasionalUse awk to sum and average a column, filtering rows by a field
Use awk to sum and average a column, filtering rows by a field
Filter with a pattern, accumulate in the body, print in END: awk '$9==200 { sum+=$10; n++ } END { if (n) print sum, sum/n }' access.log. The $9==200 pattern runs the block only on matching rows, sum+=$10 totals the bytes field, n++ counts them, and END fires once input is exhausted. Guard n so an empty match never divides by zero.
Common mistakes
- ✗Writing
$9=200(assignment, always true) instead of$9==200(comparison) - ✗Dividing by
NR(all input rows) rather than the filtered countn - ✗Printing inside the body every row instead of once in the
ENDblock
Follow-up questions
- →Why does
$9=200match every row while$9==200filters correctly? - →How would you group the totals per status code instead of just
200?
MiddleCodeOccasionalLoop over files safely when names may contain spaces or newlines
Loop over files safely when names may contain spaces or newlines
for f in $(find …) is unsafe: the substitution is word-split on IFS, so a path with a space or newline breaks into several iterations and a glob in a name expands. Iterate on NUL-delimited output instead: find … -print0 | while IFS= read -r -d '' f; do …; done. The \0 cannot occur in a filename, and IFS=/-r keep each name literal.
Common mistakes
- ✗Iterating
for f in $(find …)or$(ls), which word-splits on every space - ✗Quoting the whole substitution, which then yields one giant single value
- ✗Dropping
IFS=or-r, soreadtrims whitespace or eats backslashes
Follow-up questions
- →Why is
-print0paired withread -d ''instead of a newline delimiter? - →How would
find -exec gzip {} +compare with thewhile readloop here?
MiddleCodeOccasionalSend both stdout and stderr to one log — the 2>&1 ordering trap
Send both stdout and stderr to one log — the 2>&1 ordering trap
Redirections apply left to right, so myjob 2>&1 > run.log first points stderr at wherever stdout is now (the terminal), then moves stdout to the file — errors escape. Put the file first: myjob > run.log 2>&1. Discard errors with 2>/dev/null; feed literal stdin with a here-doc, cmd <<'EOF' … EOF (quoted tag = no expansion).
Common mistakes
- ✗Writing
2>&1before the file, so stderr copies the terminal, not the file - ✗Thinking
2>&1swaps the descriptors rather than duplicating one onto the other - ✗Believing the shell resolves redirections all at once instead of left to right
Follow-up questions
- →What does the shorthand
&>(or&>>) do, and why prefer the explicit form? - →Why does a quoted here-doc tag like
<<'EOF'disable variable expansion inside it?
MiddleCodeOccasionalCount unique IPs in an access log where the IP is the first field
Count unique IPs in an access log where the IP is the first field
Extract the first field, collapse to distinct values, and count: awk '{print $1}' access.log | sort -u | wc -l. sort -u deduplicates in one pass and wc -l counts the remaining lines; ... | sort | uniq | wc -l is equivalent. For per-IP counts use sort | uniq -c | sort -rn. The cost is dominated by the sort at O(n log n).
Common mistakes
- ✗Using
uniqwithout a precedingsort, so non-adjacent duplicates survive - ✗Counting raw lines instead of distinct first-field values
- ✗Extracting the wrong field — the IP is
$1, not$2
Follow-up questions
- →How would you change the pipeline to list the top 10 busiest IPs?
- →Why is
sort -uequivalent tosort | uniqhere, and when do they differ?
MiddleCodeOccasionalWhat does a two-stage awk pipeline over ps -ef print?
What does a two-stage awk pipeline over ps -ef print?
It prints the PID column. The first awk joins field 2 (PID) and field 8 (CMD) as PID:CMD. The second awk -F: splits each line on : and prints $1, the part before the first colon — the PID. Because the command can contain colons, only the first colon is the split boundary, so $1 is still the PID. For example 1234:/usr/bin/sshd collapses to 1234.
Common mistakes
- ✗Miscounting
ps -effields —$2is PID, not PPID - ✗Thinking
-F:keeps the part after the colon instead of$1before it - ✗Forgetting only the first
:splits, so a command with colons is safe
Follow-up questions
- →How would you change the pipeline to print the command instead of the PID?
- →Why is piping
psoutput toawkfragile if a column value contains spaces?
MiddleCodeOccasionalWhen do you rewrite a fragile bash pipeline in Python, and how?
When do you rewrite a fragile bash pipeline in Python, and how?
Switch to Python when the task outgrows line-oriented text — structured output like JSON, nested data, real error handling, or logic you want to unit-test. Here the hand-rolled JSON (trailing comma, no escaping) is the tell. Rewrite: read the log, tally a dict (a Counter), and emit with json.dumps, so quoting and structure come out correct.
Common mistakes
- ✗Hand-building JSON by string concatenation instead of
json.dumps - ✗Assuming bash is always the right tool because it is shorter to type
- ✗Ignoring that unquoted fields and a bare
grepbreak underpipefail
Follow-up questions
- →Why is
json.dumpssafer than assembling the JSON string field by field? - →How does a
grepmatching nothing interact withset -o pipefail?
MiddleCodeOccasionalFind the K most frequent IPs in a huge log in O(n log K)
Find the K most frequent IPs in a huge log in O(n log K)
Count frequencies in a hash map in one O(N) pass, then keep a min-heap of size K over the counts: push each (count, ip), and once the heap exceeds K pop the smallest. Each heap op is O(log K), so selecting over U distinct IPs is O(U log K) ≤ O(N log K). The heap leaves the K largest; sort just those K for the final order. This beats sorting all uniques at O(U log U).
Open full question →Common mistakes
- ✗Using a max-heap of size K instead of a min-heap — you must pop the smallest
- ✗Sorting all unique IPs (O(U log U)) when only the top K are needed
- ✗Forgetting the O(N) hash-count pass and trying to heap raw lines directly
Follow-up questions
- →Why does keeping the K largest require a min-heap rather than a max-heap?
- →How would you adapt this when even the distinct-IP counts exceed memory?
