Pipes & Redirection

The single richest feature in shell. Once you understand file descriptors and redirection operators, you can chain tools together in ways no other language makes this convenient.

File descriptors

Every UNIX process starts with three open file descriptors:

fd	name	default destination
0	stdin	the terminal keyboard
1	stdout	the terminal display
2	stderr	the terminal display (separate stream)

Higher fds (3, 4, 5, …) are available for your own use. The shell's redirection operators manipulate which fds connect to what.

Output redirection

cmd > out.log              # stdout → file (truncate)
cmd >> out.log             # stdout → file (append)
cmd 2> err.log             # stderr → file
cmd 2>> err.log            # stderr → file (append)
cmd > out.log 2>&1         # stdout → file, then stderr → wherever-stdout-points (= file)
cmd &> out.log             # bash shorthand: both stdout and stderr → file
cmd &>> out.log            # bash shorthand: both, append

The order matters in the POSIX form:

# WORKS — both go to file:
cmd > out.log 2>&1

# DOES NOT WORK — stderr → terminal, stdout → file:
cmd 2>&1 > out.log

Read it left-to-right: 2>&1 says "stderr now points where stdout points (currently terminal)", then > out.log says "stdout now points to file". stderr is still pointing to terminal.

The &> bash shorthand sidesteps this by being atomic.

Input redirection

cmd < input.txt            # stdin reads from file
cmd <<EOF                  # heredoc — multi-line literal stdin
hello
world
EOF
cmd <<<"$value"            # here-string — single string as stdin

Heredocs in detail

cat <<EOF
hello, $USER
the date is $(date)
EOF

Variables and $(...) expand inside heredocs by default. To suppress expansion (literal text), quote the delimiter:

cat <<'EOF'
this is $literal — no expansion
EOF

<<- strips leading tabs (not spaces) from each line, useful for indenting heredocs inside functions:

my_func() {
  cat <<-EOF
	hello
	world
	EOF
}

Here-strings

<<< feeds a single string as stdin:

read -r first second <<< "$line"           # parse a line into vars
grep -q "ERROR" <<< "$buffer"              # grep a string variable
bc <<< "scale=2; 22/7"                     # bc with one expression

Equivalent to printf '%s' "$value" | cmd but no fork — and the value is a literal string, not subject to word splitting.

Pipes

cmd1 | cmd2                # stdout of cmd1 → stdin of cmd2
cmd1 | cmd2 | cmd3         # chain
cmd |& cmd2                # bash: 2>&1 |  — pipe stderr too

Pipes are concurrent: cmd1 and cmd2 start at the same time and cmd1's output is consumed as it's produced. Don't think "cmd1 finishes, cmd2 starts" — that's wrong.

A pipeline's exit code is the last command's exit by default. Most of the time you want set -o pipefail so the first non-zero exit propagates:

set -o pipefail
curl -sf http://api/x | jq '.value'        # without pipefail, curl failures get swallowed

tee — split a stream

cmd | tee output.log                       # stdout → output.log AND stdout
cmd | tee -a output.log                    # append
cmd | tee output.log | next-stage          # log AND continue piping
cmd | tee >(grep ERROR > errors.log)       # tee + process substitution: log to errors.log too

The classic sudo + redirect problem:

sudo echo "hello" > /etc/x                 # ❌ FAILS — sudo runs echo, but redirect happens
                                           #    in your shell, which lacks write permission
echo "hello" | sudo tee /etc/x             # ✅ tee runs as root; redirects via root
echo "hello" | sudo tee /etc/x > /dev/null # …add > /dev/null to suppress tee's stdout copy

Process substitution

<(cmd) and >(cmd) make a command's stdout (or stdin) look like a file. Bash creates a named fd (e.g. /dev/fd/63) and substitutes its path into the argument list.

diff <(sort a.txt) <(sort b.txt)           # diff two sorted versions, no temp files
comm -23 <(sort a.txt) <(sort b.txt)       # lines unique to a.txt
join <(sort -t, -k1 a.csv) <(sort -t, -k1 b.csv)

Or the inverse — feed a stream INTO a command that wants a file argument:

gpg --decrypt > >(jq '.data' > parsed.json)

Process substitution is bash and zsh; not POSIX sh.

Closing fds

cmd 2>&-                                   # close stderr
cmd <&-                                    # close stdin
exec 3>&-                                  # close fd 3 in current shell

Mostly relevant when juggling custom fds for IPC or for lock files.

Custom fds

Bash supports user fds 3–9 (and arbitrarily high in modern bash):

exec 3<config.txt                          # open fd 3 for reading config.txt
read -r line <&3                           # read one line from fd 3
exec 3<&-                                  # close fd 3

# Two-stream loop without fighting over stdin:
while IFS= read -u 3 line; do
  process_line "$line"
  read -r answer                            # interactive prompt — uses default stdin
done 3< input.txt

-u N makes read use fd N. Useful when a loop both reads from a file and prompts the user.

/dev/null and /dev/stdin

cmd > /dev/null                            # discard stdout
cmd 2> /dev/null                           # discard stderr
cmd &> /dev/null                           # discard both
cat - file < /dev/stdin                    # explicitly include stdin in arg list

/dev/null is the bit bucket; everything written there is discarded. Use it to silence noisy commands.

Combining redirection patterns

{ cmd1; cmd2; cmd3; } > combined.log       # group output from multiple commands
exec > out.log 2>&1                         # redirect ALL further output of THIS shell
exec 3>&1; cmd 2>&3 | filter; exec 3>&-     # crossover: stderr to original stdout, stdout filtered

The exec form (no command after) modifies the current shell's fds. Useful at the top of a script:

exec > >(tee -a "$LOG_FILE")               # everything we print also goes to log file
exec 2>&1                                  # …and merge stderr in too

After this, every subsequent command in the script logs automatically.

Bash vs zsh

zsh has all of the same operators plus a few:

• |& is bash & zsh; same meaning (2>&1 |).
• >! in zsh forces overwrite even with noclobber set (bash uses >|).
• MULTIOS (zsh-only): cmd > a > b writes stdout to BOTH files (bash uses tee). setopt MULTIOS enables it.
• Process substitution and heredocs are identical in both.

Common bugs

cmd 2>&1 > file order bug. stderr goes to the terminal, stdout to the file. Use &> (bash) or write > file 2>&1 (correct order).

Forgetting pipefail. false | true; echo "$?" prints 0. With set -o pipefail, it prints 1. Without, your set -e doesn't catch failures inside pipelines.

sudo > file permission denied. The redirect is your shell's, not sudo's. Use | sudo tee file > /dev/null.

$(cat file) instead of $(<file). Fork-free file read is $(<file) — bash builtin. $(cat file) is two extra processes.

Heredoc with unquoted delimiter. cat <<EOF expands $vars and $(...) inside. If you want literal text (config templates, code snippets), use cat <<'EOF'. Forgetting this leaks environment values.

Capturing only stdout when you wanted both. output=$(cmd) captures stdout only. output=$(cmd 2>&1) captures both.