Pipes, I/O & jq

Every Unix process gets three standard file descriptors when it starts. Not metaphorically — literally three open files the kernel hands the process at startup.

Analogy

Every process starts its shift with three hoses already plugged in: one coming in (stdin) where raw material arrives, one going out the front (stdout) with finished product, and one going out the side (stderr) for spills and complaints. By default all three are hooked up to the terminal, but you can unplug any of them and route it somewhere else — to a file, to the trash, to the inlet of the next machine on the line. A pipe | is just welding machine A's front hose directly to machine B's inlet so the material never touches a bucket. jq is a specialist station that only accepts JSON-shaped material and reshapes it before sending it down the line.

File descriptors 0, 1, 2

FD	Name	Default destination
0	stdin	Keyboard (terminal)
1	stdout	Terminal screen
2	stderr	Terminal screen

They're file descriptors. You can redirect them anywhere.

Redirection

cmd > out.txt          # stdout to file (overwrite)
cmd >> out.txt         # stdout to file (append)
cmd 2> err.txt         # stderr to file
cmd > out.txt 2>&1     # stdout and stderr to same file
cmd < input.txt        # stdin from file
cmd 2>/dev/null        # discard stderr entirely

2>&1 reads as "redirect fd 2 to wherever fd 1 currently points". Order matters: cmd 2>&1 > out.txt redirects stderr to the original stdout (terminal), then stdout to the file — almost certainly not what you want.

Pipes

A pipe | connects stdout of one process to stdin of the next. Both processes run simultaneously:

curl -s https://api.example.com/data | jq '.users | length'

tee splits a stream: it writes to a file and passes through to stdout.

curl -s url | tee raw.json | jq '.name'

Process substitution

<(cmd) makes the output of cmd look like a file to another command:

diff <(sort a.txt) <(sort b.txt)

Without process substitution you'd have to write two temp files, diff them, and delete them.

jq — the essential JSON filter

jq reads JSON on stdin, applies a filter, and writes JSON on stdout. Its filters compose with | just like shell pipes.

The core filter set

Filter	What it does
.	Identity — pass through unchanged
.foo	Field access
.foo.bar	Nested field access
.[]	Iterate array or object values
.[0]	Array index
|	Pipe: pass left output as right input
select(pred)	Keep only values where pred is truthy
map(expr)	Apply expr to each element, return array
length	Length of string, array, or object
keys	Sorted array of object keys
type	Type name: "null", "boolean", "number", "string", "array", "object"

Examples

# Extract a field
echo '{"name":"alice","age":30}' | jq '.name'
# "alice"

# Iterate and filter
echo '[{"id":1,"active":true},{"id":2,"active":false}]' \
  | jq '.[] | select(.active) | .id'
# 1

# Reshape with map
echo '[{"id":1},{"id":2}]' | jq 'map(.id)'
# [1,2]

# Pretty-print with colour
curl -s https://api.github.com/users/octocat | jq '.'

When to reach for awk vs jq vs sed

Tool	Best for
sed	Line-by-line text substitution, simple regex replacements
awk	Columnar data, arithmetic, custom field separators
jq	Any JSON input — don't parse JSON with sed or awk
Python/Node	Complex logic that outgrows one-liners

The rule: if your input is JSON, use jq. Full stop. Parsing JSON with grep or sed breaks on nested values and whitespace variations.