Power Up Your Command Line Text Processing with Sed and Awk

Introduction

If you work with text files on the command line, you might be familiar with `sed` and `awk`. Both tools are incredibly powerful and can help you manipulate and transform text in a variety of ways. In this article, we'll explore some of the most useful features of sed and awk and show you how to use them together to achieve even more impressive results.

Image credits: Julia Evans @ bork

Getting Started with Sed

Sed (short for stream editor) is a command-line tool for processing text. It can be used to search for and replace text, delete lines that match a pattern, insert text before or after a pattern, and more. Here are a few examples of how you can use sed:

Search and Replace Text

To replace all occurrences of "old" with "new" in a file, use the following command:

$ sed 's/old/new/g' file.txt

Delete Lines that Match a Pattern: To delete all lines that contain the word "pattern", use the following command:

$ sed '/pattern/d' file.txt

To append the text "new text" after the line that contains the word "pattern", use the following command:

$ sed '/pattern/a new text' file.txt

Using Awk for Advanced Text Processing

Image Credits: Julia Evans @ bork

Awk is a powerful text-processing tool that allows you to perform complex operations on text files. It is especially useful for working with structured data like CSV files. Here are a few examples of how you can use awk:

Extract Specific Columns from a CSV File:

To extract the first and third columns of a CSV file, use the following command:

$ awk -F ',' '{print $1,$3}' file.csv

Calculate the Average of a Column in a CSV File:

To calculate the average of the first column of a CSV file, use the following command:

$ awk '{sum+=$1} END {print sum/NR}' file.csv

Find the Longest Line in a File:

To find the longest line in a file, use the following command:

$ awk '{ if (length > max) {max = length; longest = $0}} END {print longest}' file.txt

Using Sed and Awk Together

Sed and awk can be used together to achieve even more powerful text-processing capabilities. For example, you can use sed to preprocess a file and then use awk to extract specific columns or perform calculations. Here's an example:

$ sed 's/\s\+/,/g' file.txt | awk -F ',' '{print $1,$3}'

In this example, sed is used to replace all whitespace with commas, and then awk is used to extract the first and third columns of the resulting CSV file.

Conclusion

Sed and awk are powerful tools for processing text on the command line. By learning how to use them together, you can achieve even more impressive results. Whether you're working with structured data or unstructured text, sed and awk can help you streamline your workflow and get things done faster.

As you can see, sed and awk have a lot of capabilities, and this article just scratches the surface. For more advanced features, be sure to check out the official documentation for each tool. With a little practice, you'll be a sed and awk pro in no time!