Count unique IPs in an access log where the IP is the first field
You have an nginx-style access.log: one request per line, fields separated by spaces, the client IP in the first field. Print the number of distinct IPs that appear in the file.
Constraints: use only standard shell tools (awk, sort, uniq, wc); do not write a program in another language. Assume the file does not fit comfortably in one editor but fits on disk.
# access.log: "203.0.113.7 - - [10/Oct/2024:13:55:36 +0000] \"GET / HTTP/1.1\" 200"
# your pipeline hereWrite the one-liner.
Extract the first field, collapse to distinct values, and count: awk '{print $1}' access.log | sort -u | wc -l. sort -u deduplicates in one pass and wc -l counts the remaining lines; ... | sort | uniq | wc -l is equivalent. For per-IP counts use sort | uniq -c | sort -rn. The cost is dominated by the sort at O(n log n).
- ✗Using
uniqwithout a precedingsort, so non-adjacent duplicates survive - ✗Counting raw lines instead of distinct first-field values
- ✗Extracting the wrong field — the IP is
$1, not$2
- →How would you change the pipeline to list the top 10 busiest IPs?
- →Why is
sort -uequivalent tosort | uniqhere, and when do they differ?
Solution
The IP is the first whitespace-separated field, so extract it, reduce to the set of distinct values, and count what remains:
awk '{print $1}' access.log | sort -u | wc -ldistinct set in one pass. (sort | uniq is the same; uniq alone is wrong because it only removes adjacent duplicates.)
awk '{print $1}'— emits just the client IP from each line.sort -u— sorts and keeps only the first of each run of equal lines, i.e. thewc -l— counts the remaining lines = the number of distinct IPs.
To rank IPs by request volume instead of just counting them:
awk '{print $1}' access.log | sort | uniq -c | sort -rn | headuniq -c prefixes each IP with its count; sort -rn orders by that count descending. The dominant cost is the sort at O(n log n) in the number of lines.
