Lesson 6. awk and sed Where Needed

Mission link: This is stage 5, a single-lesson stage. Every prior stage stayed inside bash's own built-in capabilities; this lesson is the one place the mission deliberately reaches outside them, and the discipline of doing so only when the task actually calls for it.
Primary source: Docs: "A Sed and Awk Micro-Primer", Advanced Bash-Scripting Guide, TLDP
Prerequisites: Lesson 5, Word splitting

Warm-up

1. ▢ Why can a script that assumes bash but uses #!/bin/sh fail silently on some systems?

2. ▢ Name two bash-only features with no direct POSIX sh equivalent.

Know this

sed's shape: a stream editor for line-based, find-and-transform tasks

sed (stream editor) reads input line by line and applies simple, pattern-based edits: substitution (s/pattern/replacement/), deletion of matching lines, or operating on a specific line range. It fits precisely the shape of "find this pattern in each line and change it" or "keep only lines matching (or not matching) this pattern," and its -i flag can edit a file in place, which is why it shows up constantly in shell scripts doing lightweight text substitution (rewriting a config value, stripping comments, normalizing line endings) without needing a full programming construct for the job.

awk's shape: field-oriented processing, not just text substitution

awk is a small, complete programming language built around a pattern-action model: for each input line, it checks a pattern (often a regex, but can be any condition) and, if it matches, runs an action, with built-in support for splitting a line into fields ($1, $2, ...) on a delimiter, arithmetic, and associative arrays. This fits a materially different shape of task than sed: extracting or computing something from structured, delimited data (summing a column of numbers, printing the third field of every matching line, counting occurrences), not just transforming text in place. Reaching for sed to do column-based extraction, or awk for a one-line substitution, usually signals the wrong tool for the shape of the actual task.

The discipline: called from bash, not replacing bash's own logic

The mission's constraint is specific: awk/sed are touched only where a script genuinely needs the text-processing capability neither bash's own string operations nor a simple loop provide efficiently, not used as the default way to process every line of output. Bash itself remains the right tool for control flow (loops, conditionals, calling other programs and gluing their output together); sed/awk are invoked from a bash script for the specific line-editing or field-extraction step, then control returns to bash. A script that does everything through nested awk scripts glued together with bash is usually a sign the actual task has outgrown shell in the direction lesson 7 covers, not a sign awk was the right choice throughout.

Recognizing when neither is actually the right call

Some tasks that look like "reach for sed/awk" are better served by bash's own built-in parameter expansion (${var/pattern/replacement}, bash-only per lesson 5, or case pattern matching) when the data is a single variable rather than a stream of lines, avoiding the overhead and subprocess cost of invoking an external tool for something bash can already do internally. The judgment call this lesson asks for isn't "always prefer sed/awk over bash string handling" or the reverse; it's matching the actual shape of the task (a single value versus a stream of lines or records) to the tool actually built for that shape.

Practice

1. ▢ A script needs to replace every occurrence of old-value with new-value in a config file. Which tool fits, and why?

2. ▢ A script needs to sum the values in the third column of a CSV file. Which tool fits, and why would the other one be an awkward choice?

3. ▢ Why does the mission's constraint describe awk/sed as touched "only where a script genuinely needs" them, rather than treating them as a default text-processing habit?

4. ▢ A script needs to replace a substring in a single variable's value, not a file or stream of lines. Why might bash's own ${var/pattern/replacement} be the better fit than piping the value through sed?

5. ▢ Which claim correctly describes choosing between sed, awk, and bash's own capabilities?

   • a) awk should always be preferred over sed, since it's a more complete language
   • b) Match the tool to the actual shape of the task: sed for line-based pattern substitution, awk for field-oriented extraction or computation, and bash's own string handling for a single value that doesn't need stream processing at all
   • c) sed and awk should replace bash's control flow (loops, conditionals) wherever possible for consistency
   • d) Using sed or awk at all indicates a script has outgrown shell and should be rewritten in Python

Real-world reps

• [ ] Find a shell script you have access to that uses sed or awk. Check whether the tool used actually matches the task's shape (line substitution for sed, field extraction or computation for awk), or whether the other tool (or bash's own string handling) would have been a better fit.
• [ ] Find a place in that same script (or one you know of) where bash's own string operations or a loop handle something that's instead being piped through sed or awk unnecessarily, and consider what removing that dependency would look like.
• [ ] Tomorrow: read the primary source's brief introduction to both tools in full, and write one sentence distinguishing the shape of task each was actually designed for.

Going further

• Docs: "A Sed and Awk Micro-Primer", Advanced Bash-Scripting Guide, TLDP
• Site: "Bash Pitfalls", Greg's Wiki
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.