How CLI Redirection and Pipes Tie Together Linux Programs
Linux has a lot of built-in command-line capabilities that desktop users frequently overlook. Pop open a terminal and fire up a few standalone command-line programs to do things like finding files, monitoring system performance and automatically kick off system maintenance tasks at a certain time.
Old-time gurus sometimes reflect on the “Unix philosophy.” Linux follows that ethos. The idea is to make small, modular, do-one-job-well command-line programs that can be connected together to accomplish bigger tasks. Linux command-line programs typically are small and standalone. The notion of redirection and pipes are the links that tie programs together.
That’s what we’ll discuss today.
Redirection
Redirection in Linux refers to sending the standard output of a program to a file. The file might serve as a record of the program’s output, to log data or to analyze and/or use the data at another time. I frequently use Linux redirection to capture input from a sensor connected to a USB port on my Linux notebook or Raspberry Pi when testing of one of my gadgets.
Redirection is performed using an arrow symbol, at the command line or in a script and is pretty easy to use. Here’s an example.
Say you want to print out the names of all the files in a directory. Just type ls at the command line, while in your chosen directory.
robnotebook% ls
Maybe you want to go further and save those names in a file that you can later send to your laser printer, giving you a physical record of your files. Just send the output of the ls command to a file using the right arrow (>) symbol.
robnotebook% ls > filelog.txt
You can view the file contents with cat and print the file whenever.
robnotebook% cat filelog.txt
Simple file name listing using ls
The redirection above creates a new file named filelog.txt with the listing of directory inside. If the file didn’t exist, it is created. If the file existed already, it is overwritten with the new version of the file. You won’t get a warning about overwriting the file, so it’s best to stay focused on your work. I’ve overwritten a few “valuable” files in my day due to inattention. It likely will happen to you too. Chalk it up to experience and proceed on with things. Hopefully you’ll not be logged in as root at the time.
You don’t always have to overwrite the file. Maybe you want to add data to an existing file. Use the double arrow (>>) symbol.
A little more complicated example is collecting files from several directories.
Say you already listed your current directory (/home/rob/test) and redirected it to a file (filelog.txt).
robnotebook% ls /home/rob/test > filelog.txt
There is a sub-directory named important-files that you want to add to your existing file listing. Add the latest file listing with the following.
robnotebook% ls /home/rob/test/important-files >> filelog.txt
This time we use the full pathname and the double arrow symbol to add the output to the file listing.
ls with file redirection (add)
Redirecting program output to a file is very useful for diagnosing problems and creating data sets. Next, we’ll talk about connecting programs together.
Pipes
Another type of redirection is called a pipe. Pipes take the output of one program and send it as input to another program. Pipes connect simple command-line programs together to perform more complex tasks.
Pipes are pretty straightforward to use. Much like the right arrow for redirecting to a file, you designate a pipe with the vertical bar (|) symbol between your programs. The following is a simple example that looks for the word “and”, using grep, in all the files within the current directory and then shows one page at a time using the more command. You can use the space bar to page through the listing.
robnotebook% grep and * | more
I use the history command quite a bit, especially when developing various hardware gadgets. During testing I’ll run through a general list of tests that would be quite tedious to execute if I had to retype everything each time. My solution is to use history with the grep command to find a command line I can use as-is or make a tweak for the new situation.
For example, suppose I’m shuffling files back and forth from my Linux notebook to one of my remote Raspberry Pi projects. I use the scp command to transfer the files. Recalling the command and tweaking the file names is much easier than retyping the whole command line each time I move a file. It’s pretty painless to use a pipe to get a list of the recently used scp commands, from history. I can then select one that suites my needs.
robnotebook% history | grep scp
History with a pipe to grep
It is a simple matter then to just mod one of the command lines with the new file names and hit return. Poof, the new files are copied with a minimum of effort.
Another way I use pipes, is with the sort command.
Maybe I’m looking for large files in my current directory. You could certainly use ls -l to show all the file sizes and then just scan through the list, looking at column 5, to find the largest number. Not too efficient, beyond a few screenfuls of files, though.
robnotebook% ls -l
File listing with -l option to show file sizes
An easy way to find the largest file size is to use the sort command with a pipe.
robnotebook% ls -l | sort -n -k 5
Finding the largest file with a pipe to the sort command
Here we use ls with the -l (long) option, then pipe it through sort. The -n -k 5 means that it will look at column 5 (-k option) sorting in numerical order (-n option).
Wrap Up
We’ve seen how to send output of a program to a file and how to connect standalone command-line programs together to perform integrated tasks.
Take a look at the options to ls, sort, grep, more and so on to see how you might use these, in combination to get things done in your own Linux computing environment. The web has plenty of examples on Linux (or Unix) file redirection and pipes. With a little practice and experience, I think you’ll find the capabilities quite useful.
Contact Rob “drtorq” Reilly for consultation, speaking engagements and commissioned projects at doc@drtorq.com or 407-718-3274.
The Linux Foundation is a sponsor of The New Stack.
Feature image by 2427999 from Pixabay
