% Linux Installation and Getting Started    -*- TeX -*-
% plumbing.tex
% Copyright (c) 1992, 1993 by Matt Welsh, Larry Greenfield and Karl Fogel
%
% This file is freely redistributable, but you must preserve this copyright 
% notice on all copies, and it must be distributed only as part of "Linux 
% Installation and Getting Started". This file's use is covered by
% the copyright for the entire document, in the file "copyright.tex".

\section{UNIX Plumbing} \label{sec-plumbing}

\subsection{Standard input and output}
\index{standard input|(}
\index{standard output|(}
\index{stdin}
\index{stdout}
Many UNIX commands get input from what is known as {\bf standard input}
and send their output to {\bf standard output} (often abbreviated as ``stdin'' 
and ``stdout''). Your shell sets things up so that standard input is
your keyboard, and standard output is the screen. 

Here's an example using the command {\tt cat}. Normally, {\tt cat} reads data 
from all of the filenames given on the command line and sends this data 
directly to stdout. Therefore, using the command
\begin{tscreen}
/home/larry/papers\# {\em cat history-final masters-thesis}
\end{tscreen}
will display the contents of the file {\tt history-final} followed by
{\tt masters-thesis}. 

However, if no filenames are given to {\tt cat} as parameters, it 
instead reads data from stdin, and sends it back to stdout. Here's an
example. 
\begin{tscreen}
/home/larry/papers\# {\em cat} \\
{\em Hello there.} \\
Hello there. \\
{\em Bye.} \\
Bye. \\
\key{ctrl-D} \\
/home/larry/papers\# 
\end{tscreen}
\index{end-of-text signal}
\index{EOT (end of text) signal}
As you can see, each line that the user types (displayed in italics)
is immediately echoed back by the {\tt cat} command. When reading from
standard input, commands know that the input is ``finished'' when they
receive an EOT (end-of-text) signal. In general, this is generated by pressing
\key{ctrl-D}.

% By the way, guys, there's no such thing as an EOF character in UNIX. The
% terminal signal to signal EOT (end of text) is usually ^D. Files on
% disk don't have a terminating EOF as MS-DOS files do. The data just
% ends, and the read() call signals the end of data. :) --mdw

Here's another example. The command {\tt sort} reads in lines of
text (again, from stdin, unless files are given on the command line),
and sends the sorted output to stdout. Try the following.
\begin{tscreen}
/home/larry/papers\# {\em sort} \\
{\em bananas} \\
{\em carrots} \\
{\em apples} \\
\key{ctrl-D} \\
apples \\
bananas \\
carrots \\
/home/larry/papers\# 
\end{tscreen}
Now we can alphabetize our shopping list... isn't UNIX useful? 

\subsection{Redirecting input and output}
\index{redirection!standard input}
\index{redirection!standard output}
\index{output!redirecting}
\index{standard output!redirecting}
\index{>@{\tt >}}
Now, let's say that we wanted to send the output of {\tt sort} to a
file, to save our shopping list elsewhere. The shell allows us to
{\bf redirect} standard output to a filename, using the ``{\tt >}''
symbol. Here's how it works.
\begin{tscreen}
/home/larry/papers\# {\em sort $>$ shopping-list} \\
{\em bananas} \\
{\em carrots} \\
{\em apples} \\
\key{ctrl-D} \\
/home/larry/papers\# 
\end{tscreen}
As you can see, the result of the {\tt sort} command isn't 
displayed, instead it's saved to the file {\tt shopping-list}. 
Let's look at this file.
\begin{tscreen}
/home/larry/papers\# {\em cat shopping-list} \\
apples \\
bananas \\
carrots \\
/home/larry/papers\# 
\end{tscreen}
Now we can sort our shopping list, and save it, too!
But let's suppose that we were storing our unsorted, original shopping
list in the file {\tt items}. One way of sorting the information
and saving it to a file would be to give {\tt sort} the name of the
file to read, in lieu of standard input, and redirect standard output
as we did above. As so:
\begin{tscreen}
/home/larry/papers\# {\em sort items $>$ shopping-list} \\
/home/larry/papers\# {\em cat shopping-list} \\
apples \\
bananas \\
carrots \\
/home/larry/papers\# 
\end{tscreen}
\index{input!redirecting}
\index{standard input!redirecting}
\index{<@{\tt <}}
However, there's another way of doing this. Not only can we
redirect standard output, but we can redirect standard {\em input\/}
as well, using the ``{\tt <}'' symbol.
\begin{tscreen}
/home/larry/papers\# {\em sort $<$ items} \\
apples \\
bananas \\
carrots\\
/home/larry/papers\# 
\end{tscreen}
Technically, {\tt sort < items} is equivalent to {\tt sort items}, but
the former allows us to demonstrate the point: {\tt sort < items}
behaves as if the data in the file {\tt items} was typed to
standard input. The shell handles the redirection. {\tt sort} wasn't given 
the name of the file ({\tt items}) to read; as far as {\tt sort} is
concerned, it was still reading from standard input as if you had typed
the data from your keyboard.

\index{filter!defined}
This introduces the concept of a {\bf filter}. A filter is a program
which reads data from standard input, processes it in some way, and
sends the processed data to standard output. Using redirection,
standard input and/or standard output can be referenced from files. 
{\tt sort} is a simple filter: it sorts the incoming data and sends the
result to standard output. {\tt cat} is even simpler: it doesn't do
anything with the incoming data, it simply outputs whatever was given to it.

\subsection{Using pipes}

\index{pipes!using|(}
We've already demonstrated how to use {\tt sort} as a filter. However,
these examples assumed that you had data in a file somewhere, or were
willing to type the data to standard input yourself. What if the
data you wanted to sort came from the output of another command, such
as {\tt ls}? For example, using the {\tt -r} option with {\tt sort}
sorts the data in reverse-alphabetical order. If you wanted to list
the files in your current directory in reverse order, one way to
do it would be:
\begin{tscreen}
/home/larry/papers\# {\em ls} \\
english-list \\
history-final \\
masters-thesis \\
notes \\
/home/larry/papers\# {\em ls $>$ file-list} \\
/home/larry/papers\# {\em sort -r file-list} \\
notes \\
masters-thesis \\
history-final \\
english-list \\
/home/larry/papers\# 
\end{tscreen}
Here, we saved the output of {\tt ls} in a file, and then ran
{\tt sort -r} on that file. But this is unwieldy and causes us
to use a temporary file to save the data from {\tt ls}.

\index{pipelining!defined}
\index{pipes!creating}
The solution is to use {\bf pipelining}. Pipelining is another
feature of the shell which allows you to connect a string of
commands in a ``pipe'', where the stdout of the first
command is sent directly to the stdin of the second command, and so on.
Here, we wish to send the stdout of {\tt ls} to the stdin of {\tt sort}.
The ``{\tt |}'' symbol is used to create a pipe:
\begin{tscreen} 
/home/larry/papers\# {\em ls $\mid$ sort -r} \\
notes \\
masters-thesis \\
history-final \\
english-list \\
/home/larry/papers\# 
\end{tscreen}
This command is much shorter, and obviously easier to type. 

Another useful example---using the command
\begin{tscreen}
/home/larry/papers\# {\em ls /usr/bin} 
\end{tscreen}
is going to display a long list a files, most of which will fly
past the screen too quickly for you to read them. Instead, let's use
{\tt more} to display the list of files in {\tt /usr/bin}.
\begin{tscreen}
/home/larry/papers\# {\em ls /usr/bin $\mid$ more} 
\end{tscreen}
Now you can page down the list of files at your own leisure.

But the fun doesn't stop here! We can pipe more than two commands together.
The command {\tt head} is a filter which displays the first lines from
an input stream (here, input from a pipe). If we wanted to display
the last filename in alphabetical order in the current directory, we 
can use:
\begin{tscreen}
/home/larry/papers\# {\em ls $\mid$ sort -r $\mid$ head -1} \\
notes \\
/home/larry/papers\# 
\end{tscreen}
where {\tt head -1} simply displays the first line of input that it
receives (in this case, the stream of reverse-sorted data from {\tt ls}). 
\index{pipes!using|)}

\subsection{Non-destructive redirection}
\index{files!appeding to}
\index{redirection!non-destructive}
Using ``{\tt >}'' to redirect output to a file is destructive: in other
words, the command
\begin{tscreen}
/home/larry/papers\# {\em ls $>$ file-list}
\end{tscreen}
overwrites the contents of the file {\tt file-list}. If, instead, you
redirect with the symbol ``{\tt >>}'', the output will be appended
to the named file, instead of overwriting it.
\begin{tscreen}
/home/larry/papers\# {\em ls $>>$ file-list}
\end{tscreen}
will append the output of the {\tt ls} command to {\tt file-list}.

Just keep in mind that redirection and using pipes are features
provided by the shell---the shell provides this handy syntax using ``{\tt >}''
and ``{\tt >>}'' and ``{\tt |}''. It has nothing to do with the commands
themselves, but the shell.

\index{standard input|)}
\index{standard output|)}
% Need to cover use of << and possibly use of stderr. Problem with 
% covering stderr is that it's different in different shells. Maybe later.

