Sed is an early UNIX program meant to function as a non-interactive ‘stream editor’. It is one of the earliest programs to support the use of regular expressions for pattern matching, and has remained a popular for editing and filtering streams of text from the command line. In this article, I’m going to go through some of the basics, and provide some examples of what can be accomplished with basic sed programs.
There are many different versions of sed, with the most common being the BSD and GNU flavors. The BSD version is used by MacOS, while the GNU version comes installed on Linux machines. They are similar… yet different. Sed commands written for the BSD version will not work on the GNU version. You can find a summary of the differences between the two versions here. For this article, I will be using a MacBook.
Making changes to a file.
Let’s first get a poem, and put it into a poem.txt file. From your terminal, type the following:
You should see something like:
The last line of the poem and the prompt both print out on the same line. We can fix this with our first sed command.
Perfect, we inserted a newline at the end of the file and now it looks a little neater. Now try printing the poem again. What do you notice?
It went back to the way it was before! This is because by default sed only outputs changes to stdout, it does not modify the original file. However, you we can add an argument: -i ‘’ in order to make the changes permanently. Those quotation marks are important, they are needed for the Mac (BSD) version of sed, but throw an error on the GNU version. So if we type:
The changes we make will be permanent.
Be careful using sed to make permanent changes to files, you could very easily make a lot of irreversible changes and ruin a lot of files if you are not careful. Always test your commands first and make sure they are doing what you expect.
In the above expression,
One of the most popular uses of sed is for substituting patterns. Let’s take our poem file and make some simple substitutions.
We start the program by calling sed, then within the quotes, s stands for substitution, followed by the word we want to substitute, and what we want to substitute it with. The slashes are known as delimiters, they separate the pattern we want to find and the pattern we want to replace. Let’s try another one.
Something happened. Our substitution was made on the first occurrence of sheep (turning it into COW), but not on the second. As stated before, sed is a ‘stream editor’, meaning that it operates line-by-line, makes changes to the first match that it finds on a line, then skips all other matches and continues to the next line. We can change this by adding g — short for global — to the end of our pattern. Let’s also make these changes to our original file, because we want white cows not white sheep!
When changing the file in place, your command will not return anything. So you need to reopen your file to see any changes:
The above command is simplified. We could replace sheep with regular expressions for more complicated pattern matching.
Sed can also be used to modify spreadsheets, although, there are usually other methods that are better suited for spreadsheet manipulation. To demonstrate how you might make some basic changes to a spreadsheet, let’s load some up!
Supposed we have two files called op1.csv, and op2.csv:
We want to change the header lines of op1.csv from ‘Target Country’ and ‘Target City’, to ‘COUNTRY’ and ‘CITY’. Then, we want to combine both spreadsheets into a new, combined file called op3.csv. We can achieve this with the following:
In the above pattern we created a new document with new headers, and appended two .csv files, all in one line of code!
This article barely scratches the surface of what can be done with sed, you can find a more in depth overview of the capabilities of sed here. As I mentioned before, there are two versions of sed. If you have a Mac you can download and use the Linux (GNU) version with this command:
The first option will allow you to retain both version, so you would be able to call sed, or gsed depending on the version you would want to use. The second option will just replace sed with the GNU version so you do not need the g.
Notice that when invoking gsed we no longer needed the ‘’ after -i, and the /gi makes a global case insensitive match.