Linux & the Command Line

Pipes, Redirection and Combining Commands

30 min

Each Unix tool does one thing well. Pipes connect them: the output of one command becomes the input of the next.

Standard streams

Every program has three streams:

Stream Number Default
stdin (input) 0 keyboard
stdout (output) 1 screen
stderr (errors) 2 screen

Redirection

ls > files.txt            # stdout to a file (overwrite)
ls >> files.txt           # append
ls /nope 2> errors.txt    # only errors to a file
cmd > out.log 2>&1        # both output and errors to one file
cmd > /dev/null 2>&1      # throw everything away
sort < names.txt          # file as input

2>&1 means "send stream 2 to wherever stream 1 is going". Order matters: put it after the > redirect.

Pipes

cat server.log | grep ERROR | wc -l
ps aux | grep nginx
history | grep git | tail -5

The toolbox that makes pipes useful

Tool Does
sort sort lines (-n numeric, -r reverse)
uniq -c count repeated adjacent lines (sort first)
cut -d, -f2 take column 2 of comma-separated text
awk '{print $1}' print the first whitespace-separated field
tr a-z A-Z translate characters
head, tail first or last lines
tee file write to a file AND pass along
xargs turn input lines into arguments

Worked example: top visitors

Which IP addresses hit the server most?

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

Read it left to right: take the first field (the IP), sort so duplicates are adjacent, count them, sort by count descending, show the top 10. This one line replaces a small program.

Chaining with && and ||

mkdir build && cd build          # cd only if mkdir succeeded
make || echo "build failed"      # run echo only if make failed
cmd1 ; cmd2                      # run both regardless

Every command returns an exit code: 0 means success, anything else is failure. echo $? shows the last one. && and || decide based on it, and so do scripts and CI pipelines.

Practice

Complete the Chain commands mission: count the ERROR lines in ~/logs/server.log with a pipe.

Check yourself

  • What does 2>&1 do?
  • Why must you sort before uniq -c?
  • What is the exit code of a successful command?

Your progress is saved in this browser only.

Submit for review (10 points)