MiddleCodeOccasionalNot answered yet

When do you rewrite a fragile bash pipeline in Python, and how?

This one-liner counts ERROR lines per service and hand-builds a JSON summary, but it emits a trailing comma and no braces, does not escape service names, and silently produces nothing when grep matches zero lines under pipefail.

Constraints: the output must be valid JSON {"svc": count, …}; explain the signal that a bash pipeline should become a Python script, then rewrite it.

grep ERROR app.log | awk '{print $3}' | sort | uniq -c \
  | awk '{printf "\"%s\": %s,", $2, $1}'

State when to prefer Python and rewrite the pipeline.

Switch to Python when the task outgrows line-oriented text — structured output like JSON, nested data, real error handling, or logic you want to unit-test. Here the hand-rolled JSON (trailing comma, no escaping) is the tell. Rewrite: read the log, tally a dict (a Counter), and emit with json.dumps, so quoting and structure come out correct.

✗Hand-building JSON by string concatenation instead of json.dumps
✗Assuming bash is always the right tool because it is shorter to type
✗Ignoring that unquoted fields and a bare grep break under pipefail

→Why is json.dumps safer than assembling the JSON string field by field?
→How does a grep matching nothing interact with set -o pipefail?

Solution

The signal to switch tools. bash is great for line-oriented pipelines, but here we need structured output (JSON) with correct escaping, and under pipefail an empty grep aborts the whole pipe. Once data structures, escaping, error handling, or a wish to write tests appear, it is time for Python.

#!/usr/bin/env python3
import json
from collections import Counter

counts = Counter()
with open("app.log") as f:
    for line in f:
        if "ERROR" not in line:
            continue
        parts = line.split()
        if len(parts) >= 3:
            counts[parts[2]] += 1     # service is the third field

print(json.dumps(counts, ensure_ascii=False, sort_keys=True))

for you, with no trailing comma.

logic is easy to cover with a unit test.

Counter tallies in one pass — the sort | uniq -c idiom, but in memory.
json.dumps builds a valid object: braces, quotes, and escaping are handled
Empty input yields {} rather than silent nothing, and the split/filter

This is not "bash is bad" — for grep | wc -l it is ideal. The threshold is structure and robustness, not file size.

Open topic in roadmap →

When do you rewrite a fragile bash pipeline in Python, and how?

Constraints: the output must be valid JSON {"svc": count, …}; explain the signal that a bash pipeline should become a Python script, then rewrite it.

grep ERROR app.log | awk '{print $3}' | sort | uniq -c \
  | awk '{printf "\"%s\": %s,", $2, $1}'

State when to prefer Python and rewrite the pipeline.

Solution

#!/usr/bin/env python3 import json from collections import Counter counts = Counter() with open("app.log") as f: for line in f: if "ERROR" not in line: continue parts = line.split() if len(parts) >= 3: counts[parts[2]] += 1 # service is the third field print(json.dumps(counts, ensure_ascii=False, sort_keys=True))

for you, with no trailing comma.

logic is easy to cover with a unit test.

Counter tallies in one pass — the sort | uniq -c idiom, but in memory.

json.dumps builds a valid object: braces, quotes, and escaping are handled

Empty input yields {} rather than silent nothing, and the split/filter

This is not "bash is bad" — for grep | wc -l it is ideal. The threshold is structure and robustness, not file size.