Previous Page
Next Page

4.11. Using Shell Redirection and Piping

The Unix/Linux philosophy revolves around the concept of programs as building blockseach one intended to do one job and do it well. Redirection lets you connect these commands to files, and piping enables you to plug commands together like a child's toy.

4.11.1. How Do I Do That?

Each command has three numbered file descriptors that are opened automatically:


standard input (stdin, file descriptor 0)

The normal input to the program


standard output (stdout, file descriptor 1)

The normal output from the program


standard error (stderr, file descriptor 2)

Error messages from the program

By default, these file descriptors are connected to the terminal, if one is available, so standard input comes from the terminal keyboard, and standard output and standard error go to the terminal screen. Programs may open any other connections they need to read or write files, communicate with other local programs, or communicate with programs over the network.

Connections to the graphical user interface are created by opening a network connection from the program (client) to the X Window server. This is distinct from the three standard file descriptors.


4.11.1.1. Redirection

To redirect the output of a program to a file, use the greater-than (>) symbol followed by the name of the file:

$ cal 7 2006
     July 2006      
Su Mo Tu We Th Fr Sa
                   1
 2  3  4  5  6  7  8
 9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
$ cal 7 2006 >month.txt
$ cat month.txt     
     July 2006      
Su Mo Tu We Th Fr Sa
                   1
 2  3  4  5  6  7  8
 9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31

When you redirect output with >, the previous contents of the file are overwritten. To append (add) to the file, use >>:

$ cal 3 2009 >>month.txt
$ cat month.txt
     July 2006      
Su Mo Tu We Th Fr Sa
                   1
 2  3  4  5  6  7  8
 9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
     March 2009     
Su Mo Tu We Th Fr Sa
 1  2  3  4  5  6  7
 8  9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31

Error messages are not sent to standard output, so you can still see the messages even when standard output is redirected:

$ cal 17 2009 >month.txt
cal: illegal month value: use 1-12

To redirect error messages, place the file descriptor number (2) in front of the redirection symbol (> or >>):

$ cal 17 2009 2>errors
$ cat errors
cal: illegal month value: use 1-12

You can redirect both standard output and standard error:

$ cal 17 2009 >month.txt 2>errors
               

To redirect the input of a command, use the less-than sign (<) followed by the filename containing the data you wish to use as the input:

$ echo "2^8" >problem
$ bc <problem
256

bc is a calculator program. The first command places a numeric expression in the file problem; the second line starts bc, using problem as the input. The output from bc is the solution of the expression: 256.

Of course, you can redirect both input and output:

$ bc <problem >result

4.11.1.2. Piping

A pipe is a mechanism used to connect the standard output of one program to the standard input of another program. To create a pipe, insert the vertical-bar (|) symbol between the two commands:

$ mount
/dev/mapper/main-root on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/hdc2 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/main-home on /home type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sdb on /media/disk type vfat (rw,noexec,nosuid,nodev,shortname=winnt,uid=503)
$ mount | grep /dev/mapper
/dev/mapper/main-root on / type ext3 (rw)
/dev/mapper/main-home on /home type ext3 (rw)

In this example, the output of the mount command is used as the input to the grep command, which outputs only lines that match the specified pattern. A group of commands connected together with pipe symbols is known as a pipeline. You can extend a pipeline by connecting additional commands:

$ mount | grep /dev/mapper | sort
/dev/mapper/main-home on /home type ext3 (rw)
/dev/mapper/main-root on / type ext3 (rw)

The input to a pipeline and the output from a pipeline may be redirected:

$ cut -d: -f1 </etc/passwd|sort|head >output
$ cat output
adm
apache
avahi
beaglidx
bin
chip
chris
daemon
dbus
distcache

However, it's essential that the input redirect take place at the start of the pipeline (at the command on the left) and that the output redirection take place at the end (at the command on the right). Consider this wrong example:

$ cut -d: -f1 </etc/passwd|sort >output|head
               

In this case, it's unclear whether the standard output of sort should be placed in the file output or used as the standard input to the head command. The result is undefined (which means don't do this!).

4.11.2. How Does It Work?

Redirection is set up by the bash shell before the command is executed. If there is a redirection error (such as an invalid filename or a file permission problem), it will be reported by the shell and the command will not be executed:

$ cal >foo/bar/baz
bash: foo/bar/baz: No such file or directory

Note that the error message starts with bash, indicating that it was produced by the shell and not by the cal command.


A command is not aware of file redirection unless it has specifically been programed to check the standard file descriptors or perform special operations on them (such as changing terminal characteristics). Redirected file descriptors are inherited by applications that were started by commands; in this example, the nice command starts the cal command, and cal inherits the redirection set up for nice:

$ nice "cal" >test.txt
$ cat test.txt
     July 2006      
Su Mo Tu We Th Fr Sa
                   1
 2  3  4  5  6  7  8
 9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31

4.11.3. What About...

4.11.3.1. ...redirecting standard output and standard error to the same destination?

You can use the characters 2>&1 to redirect standard error to the same destination as standard output:

$ cal 17 2009 >/tmp/calresult 2>&1
               

Notice that the order of the redirections matters. The preceding command will redirect all output to /tmp/calresult, but this command will not redirect standard error:

$ cal 17 2009 2>&1 >/tmp/calresult

The 2>&1 redirection is evaluated first, so standard error is directed to the same destination as standard output (which, at that point, is the terminal); > /tmp/calresult then redirects standard output by itself.

This construct can also be used with piping:

$ cal 17 2009 2>&1 | head -2
               

This will feed both the standard output and the standard error from cal into the standard input of head.

4.11.3.2. ...redirecting to a device?

Linux treats most devices as files, so you can redirect data to and from devices easily. This command copies the first 50 lines of the /etc/services file directly to a parallel printer port:

$ head -50 /etc/services >/dev/lp0

4.11.3.3. ...splitting a pipe to send data to two destinations?

The tee command will receive data on standard input and write one copy to a file and one copy to standard output. This effectively splits a pipe:

$ cal -y | tee /tmp/thisyear.txt | head -2

To send a copy of the data to the screen, use tee with the device file /dev/tty (the current terminal):

$ cal -y | tee /dev/tty | grep Mo | head -1 >/tmp/dow-header.txt 
                  


4.11.3.4. ...piping and redirecting data that is not text?

No assumptions are made about the type of data being piped or redirected; in fact, there are many programs that are designed to work with piped graphics, audio, or video data streams. For example, this pipeline will decode a color JPEG image, scale it to half-size, convert it to grayscale, normalize it, convert it back into a JPEG, save a copy as /tmp/final.jpg, and display the output in a window:

$ djpeg /usr/share/wallpapers/floating-leaves.jpg | pnmscale 0.5 |       ppmtopgm | ppmnorm | cjpeg | tee /tmp/final.jpg | display -
               

4.11.4. Where Can I Learn More?

  • The manpage for bash


Previous Page
Next Page