Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Linux / Open Source

How CLI Redirection and Pipes Tie Together Linux Programs

Linux has a lot of built-in command-line capabilities that desktop users frequently overlook. Here is a set for piping the output of one job to the input of another.
Jun 8th, 2020 12:22pm by
Featued image for: How CLI Redirection and Pipes Tie Together Linux Programs

Linux has a lot of built-in command-line capabilities that desktop users frequently overlook. Pop open a terminal and fire up a few standalone command-line programs to do things like finding files, monitoring system performance and automatically kick off system maintenance tasks at a certain time.

Old-time gurus sometimes reflect on the “Unix philosophy.” Linux follows that ethos. The idea is to make small, modular, do-one-job-well command-line programs that can be connected together to accomplish bigger tasks. Linux command-line programs typically are small and standalone. The notion of redirection and pipes are the links that tie programs together.

That’s what we’ll discuss today.


Redirection in Linux refers to sending the standard output of a program to a file. The file might serve as a record of the program’s output, to log data or to analyze and/or use the data at another time. I frequently use Linux redirection to capture input from a sensor connected to a USB port on my Linux notebook or Raspberry Pi when testing of one of my gadgets.

Redirection is performed using an arrow symbol, at the command line or in a script and is pretty easy to use. Here’s an example.

Say you want to print out the names of all the files in a directory. Just type ls at the command line, while in your chosen directory.

robnotebook% ls

Maybe you want to go further and save those names in a file that you can later send to your laser printer, giving you a physical record of your files. Just send the output of the ls command to a file using the right arrow (>) symbol.

robnotebook% ls > filelog.txt

You can view the file contents with cat and print the file whenever.

robnotebook% cat filelog.txt

Simple file name listing using ls

The redirection above creates a new file named filelog.txt with the listing of directory inside. If the file didn’t exist, it is created. If the file existed already, it is overwritten with the new version of the file. You won’t get a warning about overwriting the file, so it’s best to stay focused on your work. I’ve overwritten a few “valuable” files in my day due to inattention. It likely will happen to you too. Chalk it up to experience and proceed on with things. Hopefully you’ll not be logged in as root at the time.

You don’t always have to overwrite the file. Maybe you want to add data to an existing file. Use the double arrow (>>) symbol.

A little more complicated example is collecting files from several directories.

Say you already listed your current directory (/home/rob/test) and redirected it to a file (filelog.txt).

robnotebook% ls /home/rob/test > filelog.txt

There is a sub-directory named important-files that you want to add to your existing file listing. Add the latest file listing with the following.

robnotebook% ls /home/rob/test/important-files >> filelog.txt

This time we use the full pathname and the double arrow symbol to add the output to the file listing.

ls with file redirection (add)

Redirecting program output to a file is very useful for diagnosing problems and creating data sets. Next, we’ll talk about connecting programs together.


Another type of redirection is called a pipe. Pipes take the output of one program and send it as input to another program. Pipes connect simple command-line programs together to perform more complex tasks.

Pipes are pretty straightforward to use. Much like the right arrow for redirecting to a file, you designate a pipe with the vertical bar (|) symbol between your programs. The following is a simple example that looks for the word “and”, using grep, in all the files within the current directory and then shows one page at a time using the more command. You can use the space bar to page through the listing.

robnotebook% grep and * | more

I use the history command quite a bit, especially when developing various hardware gadgets. During testing I’ll run through a general list of tests that would be quite tedious to execute if I had to retype everything each time. My solution is to use history with the grep command to find a command line I can use as-is or make a tweak for the new situation.

For example, suppose I’m shuffling files back and forth from my Linux notebook to one of my remote Raspberry Pi projects. I use the scp command to transfer the files. Recalling the command and tweaking the file names is much easier than retyping the whole command line each time I move a file. It’s pretty painless to use a pipe to get a list of the recently used scp commands, from history. I can then select one that suites my needs.

robnotebook% history | grep scp

History with a pipe to grep

It is a simple matter then to just mod one of the command lines with the new file names and hit return. Poof, the new files are copied with a minimum of effort.

Another way I use pipes, is with the sort command.

Maybe I’m looking for large files in my current directory. You could certainly use ls -l to show all the file sizes and then just scan through the list, looking at column 5, to find the largest number. Not too efficient, beyond a few screenfuls of files, though.

robnotebook% ls -l

File listing with -l option to show file sizes

An easy way to find the largest file size is to use the sort command with a pipe.

robnotebook% ls -l | sort -n -k 5

Finding the largest file with a pipe to the sort command

Here we use ls with the -l (long) option, then pipe it through sort. The -n -k 5 means that it will look at column 5 (-k option) sorting in numerical order (-n option).

Wrap Up

We’ve seen how to send output of a program to a file and how to connect standalone command-line programs together to perform integrated tasks.

Take a look at the options to ls, sort, grep, more and so on to see how you might use these, in combination to get things done in your own Linux computing environment. The web has plenty of examples on Linux (or Unix) file redirection and pipes. With a little practice and experience, I think you’ll find the capabilities quite useful.

Contact Rob “drtorq” Reilly for consultation, speaking engagements and commissioned projects at or 407-718-3274.

The Linux Foundation is a sponsor of The New Stack.

Feature image by 2427999 from Pixabay

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.