18.2 Blank Line and Empty Line Removal

20190204

A task we sometimes find useful is to remove all of the blank lines from a file. Suppose we have empty lines separating sentences and want to bring them together into a single paragraph:

cat myfile_blank_lines.txt

## Linux has many powerful command line tools dating from the 1970's.
## 
## Unix and Linux tools focus on a specific task.
## 
## Tools then freely collaborate together to accomplish the task.

We can remove blank lines using awk:

awk NF < myfile_blank_lines.txt

## Linux has many powerful command line tools dating from the 1970's.
## Unix and Linux tools focus on a specific task.
## Tools then freely collaborate together to accomplish the task.

This very simple use of awk is powerful and elegant in its simplicity. The awk variable NF is the number of fields in the input record. For blank lines this is zero. The default awk action is to print lines for which the pattern is true. Here the pattern is NF. Zero is interpreted as false and so nothing is printed.

We can save the result into a new file:

$ awk NF < myfile.txt > noblanks.txt

We could also use sed.

$ sed '/^$/d' myfile.txt

The pattern here will match any empty line since ^ matches the beginning and $ matches the end of the line and allows for no other characters in between. For lines that match the pattern the d operator is utilised to delete the line.

Then we can operate on the file in-place using -i which is short for --in-place.

$ sed -i '/^$/d' myfile.txt

Some further alternatives and interesting variations below and also see Stack Overflow.

$ sed '/^ *$/d' myfile.txt           # Blank lines may have space.
$ sed '/^[[:space:]]*$/d' myfile.txt # Or they may include tabs.
$ sed '/^\s*$/d' myfile.txt          # Short version of above.
$ grep . myfile.txt                  # Grep non-empty lines.
$ grep '\S' myfile.txt               # Grep non-space only lines.

Your donation will support ongoing availability and give you access to the PDF version of this book. Desktop Survival Guides include Data Science, GNU/Linux, and MLHub. Books available on Amazon include Data Mining with Rattle and Essentials of Data Science. Popular open source software includes rattle, wajig, and mlhub. Hosted by Togaware, a pioneer of free and open source software since 1984. Copyright © 1995-2022 Graham.Williams@togaware.com Creative Commons Attribution-ShareAlike 4.0