Five awk and sed one-liners, taken apart

Most awk and sed one-liners look like noise until you split them into pieces. Below are five that come up often. Each one gets the command, then a walk through its parts. The examples assume GNU awk and GNU sed, which is what most Linux systems ship. macOS differs in one place, covered at the end.

Print the last field of every line

awk '{print $NF}' notes.txt

• awk reads its input one line at a time and runs a program on each line.
• '{print $NF}' is the program. There is no pattern in front of the braces, so the action runs on every line.
• awk splits each line into fields on whitespace by default. $1 is the first field, and NF holds the field count. $NF therefore means "the field whose number is NF", which is the last one.
• print with no argument prints the whole line. With a field, it prints only that field.

This is handy on command output where the number of columns changes from line to line. For example, ls -l | awk '{print $NF}' lists the file names, though it will split names that contain spaces.

List usernames with UID 1000 and up

awk -F: '$3>=1000 {print $1}' /etc/passwd

• -F: sets the field separator to a colon. Each line of /etc/passwd is made of colon-separated fields.
• $3>=1000 is a pattern. A pattern placed before the braces decides whether the action runs for that line. Here it checks the third field, the UID, as a number.
• {print $1} prints the first field, the username.

Regular Linux users start at UID 1000 on most distributions, so this shows them and skips system accounts. It will also list the nobody account, which usually has UID 65534. On macOS the real accounts live in Directory Services and not in this file, so use dscl there instead.

Remove duplicate lines but keep the order

awk '!seen[$0]++' file.txt

• There is no {} action, so awk falls back to its default action, which is printing the line.
• seen is an associative array. Reading a key that has never been set gives 0.
• $0 is the whole line, and it is used as the key.
• seen[$0]++ returns the old value and then adds 1. The first time a line appears the old value is 0, so !0 is true and the line prints. Every later copy returns 1 or more, so ! makes it false and the line is skipped.

sort -u also removes duplicates, but it reorders the output. This version keeps the first occurrence where it was. The tradeoff is memory, since awk keeps every unique line in the array.

Print lines 10 through 20

sed -n '10,20p' file.txt

• sed is the stream editor. Like awk, it reads one line at a time.
• -n turns off sed's default behavior of printing every line.
• 10,20 is an address range. It matches line 10 through line 20, inclusive.
• p is the print command. It prints whatever the address selects.

Put together, -n plus p prints only the selected range. Without -n, the whole file would print, and lines 10 through 20 would appear twice. On a big file, add a quit so sed stops reading: sed -n '10,20p;20q' file.txt.

Replace text in place and keep a backup

sed -i.bak 's/old/new/g' file.txt

• -i.bak edits the file in place. The text attached to -i is a backup suffix. sed saves the original as file.txt.bak and writes the result to file.txt.
• s/old/new/ is the substitute command. old is a regular expression, and new is the replacement text.
• The trailing g flag replaces every match on each line. Without it, only the first match on each line changes.
• file.txt is the file to edit.

Remember that old is a regex, so characters such as ., * and [ have special meaning. Escape them with a backslash when you want them literal.

The macOS difference

macOS ships BSD sed, and its -i takes the backup suffix as a separate argument. An empty argument means no backup:

sed -i '' 's/old/new/g' file.txt     # edit in place, no backup
sed -i .bak 's/old/new/g' file.txt   # edit in place, keep file.txt.bak

On GNU sed, the attached form -i.bak is the right one. On BSD sed, a plain -i with no suffix takes the script as the suffix and fails with a confusing error about a missing file. Test any -i command on a copy of the file first, because it changes the original immediately.