sed, known as the “streams editor”, is a powerful tool for text manipulation on UNIX systems. Unlike vim, sed can operate on files of limitless size. The session will cover some of the intermediate concepts such as grouping and back referencing, regular expressions, replacing the nth occurrence of a pattern, operation on every nth line and so on. We will spice up the session with a few “sed one-liners/idioms”
3. What is sed
3
non-interactive
non-screen oriented
Line oriented
Input file can be any size
Input file not affected
4. sed syntax
4
sed [options] 'cmd' in_file(s)
sed [options] 'cmd' in_file(s) [> out_file]
Standard input: sed [options] 'cmd' < in_file [>
out_file]
Pipelined input: command | sed [options] 'cmd' [>
out_file]
5. Simple examples
5
Example Explanation
sed 's/pat1/pat2/g' file Substitute all occurrences of pat1 with pat2
sed '/pat1/d' file delete lines containing pat1 from file
sed '/pat1/w newfile' file save lines containing pat1 to newfile
sed –n '30,40p' file Print lines 30 to 40
sed '10q' file Print the top 10 lines
sed –e 's/pat1/pat2/' –e Substitute pat1 with pat2, and pat3 with pat4
's/pat3/pat4/' file
sed Substitute pat1 with pat2, and pat3 with pat4
's/pat1/pat2/;s/pat3/pat4
/' file
6. Different sed options/switches
6
Range Remarks
-e Used when multiple commands are used on the
command line
-n Suppress automatic printing of pattern space
-f script-file The commands in the script-file get executed
-r Use extended regular expressions in the script. With
this option, characters such as (, ), {, }, | become meta
characters, and don't have to be escaped.
-s Consider files as separate rather than as a single
continuous long stream
-i in-place editing of the input file
7. The "-f" option - example
7
cat mysed.txt
/pat1/ s/this/that/g
/pat2/ s/before/after/
sed –f mysed.txt in_file
apostrophes not used
8. With and without "-s" option -
Comparison
8
Lines Lines sed –n '1,10p' f1, f2 sed –ns '1,10p' f1, f2
in f1 in f2
4 5 9 lines 4 lines from f1, 5 lines from f2
6 6 6 lines from f1, 4 lines from f2 6 lines from f1, 6 lines from f2
12 10 10 lines from f1. No lines from 10 lines from f1, 10 lines from
f2 f2
9. Address Specification
9
Range Remarks
10 Just the line 10
1,10 Lines between 1 and 10
10,$ Line 10 to end of file
10,+3 Line 10 and 3 lines below (lines 10, 11, 12 and 13)
10~3 every third line after line 10
10, ~3 Line 10 and the next multiple of 3 (lines 10,11 and 12)
/pat1/ All lines containing pat1
/pat1/,+3 Lines containing pat1 and three lines following it
/pat1/,~3 Lines containing pat1 and up to the multiple of 3
/pat1/, 20 Lines between the line containing pat1 and line 20 if pat1 appears
before line 20. Otherwise, just the line containing pat1
/pat1/, /pat2/ lines between the line containing pat1 and line
containing pat2
10. Address Specification with negation
10
Range Remarks
10! All lines except 10 (! is the negation indicator)
1,10! Lines from 11 to end of the file
10,$! Lines 1 to 9
10,+3! All lines except 10, 11, 12 and 13
10~3! All lines except line 10 and every third line after that
10, ~3! All lines except lines 10, 11, and 12
/pat1/! All lines not containing pat1
/pat1/,+3! All line except lines containing pat1 and three lines
following it
/pat1/,~3! All lines containing pat1 and up to the multiple of 3
11. Regular Expressions
11
Meta character Meaning
. Matches any single character except newline
* Matches zero or more of the character preceding it
e.g.: bugs*, table.*
^ Denotes the beginning of the line. ^A denotes lines starting
with A
$ Denotes the end of the line. :$ denotes lines ending with :
Escape character (., *, [, , etc)
[] matches one or more characters within the brackets. e.g.
[aeiou], [a-z], [a-zA-Z], [0-9], [:alpha:], [a-z?,!]
[^] matches any characters others than the ones inside brackets.
eg. ^[^13579] denotes all lines not starting with odd numbers,
[^02468]$ denotes all lines not ending with even numbers
<, > Matches characters at the beginning or end of words
12. Extended Regular Expressions
12
Meta character Meaning
| alternation. e.g.: ho(use|me), the(y|m), (they|them)
+ one or more occurrences of previous character. a+ is same as
aa*)
? zero or one occurrences of previous character.
{n} exactly n repetitions of the previous char or group
{n,} n or more repetitions of the previous char or group
{,m} zero to m repetitions on the previous char or group
{n, m} n to m repetitions of previous char or group
(....) Used for grouping
sed –r ... the "-r" option may have to be used on some version of sed for
extended regular expressions to work
e.g.: sed –r '/(pat1|pat2) s/pat3/pat4/'
sed –rn '/(pat1|pat2){3,}/ p'
13. Regular Expressions – Examples
13
Example Meaning
.{10,} 10 or more characters. Curly braces have to
escaped
[0-9]{3}-[0-9]{2}-[0-9]{4} Social Security number
([1-9]{3})[1-9]{3}-[0-9]{4} Phone number (xxx)yyy-zzzz
[0-9]{2,3}.[0-9]{1,3}.[0- IP address format
9]{1,3}.[0-9]{1,3}
[0-9]{3}[ ]*[0-9]{3} Postal code in India
[0-9]{5}(-[0-9]{4})? US ZIP Code + 4
14. Substitution – Format and Options
14
[address[!]]s/pat1/pat2/[options]
Options: g – global, w – write, i – ignore case, p –
print, n – nth occurrence
: or ; +, @ or any other character including space
could also be used as the delimiter. Useful when / is
part of the search or replacement string.
15. Substitution - Examples
15
Example Explanation
sed 's/pat1/pat2/' fn Substitute the FIRST occurrence of pat1 with pat2
sed 's/pat1/pat2/g' fn Substitute ALL occurrences of pat1 with pat2
sed 's/pat1/pat2/3' fn Substitute the third occurrence of pat1 with pat2
sed 's/pat1/pat2/3g' fn Substitute all but the first two occurrences of pat1
with pat2
sed 's/pat1/pat2/gi' fn Substitute ALL occurrences of pat1 with pat2
ignoring the case
sed 's/pat1/pat2/giw Write to new_files lines containing pat1
new_file' fn substituting with pat2
sed –n 's/pat1/pat2/gp' fn Print lines containing pat1 substituting with pat2
16. Substitution – Examples contd
16
Example Explanation
sed '/pat1/ s/pat2/pat3/g' fn Substitute all occurrences of pat2 with pat3 on
lines containing pat1
sed '/pat1/! s/pat2/pat3/g' Substitute all occurrences of pat2 with pat3 on
fn lines NOT containing pat1
sed '/pat1/,/pat2/ Substitute all occurrences of pat3 with pat4 on
s/pat3/pat4/g' fn lines between pat1 and pat2 (inclusive)
sed '1,100 s/pat1/pat2/gi' fn Substitute ALL occurrences of pat1 with pat2
ignoring the case between lines 1 and 100
sed '/pat1/,/pat2/! Substitute all occurrences of pat3 with pat4 on
s/pat3/pat4/g' fn lines NOT between pat1 and pat2 (inclusive)
sed '1,100! s/pat1/pat2/3i' fn Substitute the third occurrence of pat1 with pat2
ignoring the case from 101 to end of file
sed '2~5 s/pat1/pat2/w Write every 5th line starting with the 2nd line,
new_file ' fn substituting pat1 with pat2
17. Substitution – Examples contd.
17
Example Explanation
sed Substitute all occurrences of either pat1 or pat2
's/(pat1|pat2)/pat3/g' fn with pat3
sed –r 's/(pat1|pat2)/pat3/g' Same as above. With the –r option, parenthesis and
fn alternation characters don't have to be escaped
sed –r Substitute all occurrences of either abcpat1 or
's/abc(pat1|pat2)/pat3/g' fn abcpat2 with pat3
sed 's/<pat1/pat3/g' fn Substitute all occurrences pat1 that begin a word
with pat3.
sed 's/pat1>/pat3/g' fn Substitute all occurrences pat1 that is end of a
word with pat3
sed 's/<pat1>/pat3/g' fn Substitute all occurrences of the word pat1 with pat3.
Note: The angular brackets (< and >) have to be escaped
even with the –r option.
sed 's/[a-z]/u&/g' fn Substitute all lower case letters to upper case letters
sed 's/./&&/g' fn Double each character
18. Grouping and Back Referencing
18
Parts of strings in the Search/Left-hand side can be
grouped and referenced in the Replacement/Right-
hand side
Up to nine groups possible (1, 2, ..9)
Groups can be nested or referenced back on the
Search side
Same group can be referenced any number of times
19. Substitution with Grouping and Back
Referencing. Examples
19
Command Explanation
sed 's/^(.*):(.*)/2:1/' fn Swap two fields delimited with :. "column
A:column B" becomes "column B:column A"
sed –r 's/^(.*):(.*)/2:1/' fn Same as above. With the –r option, the
parentheses don't have to be escaped.
sed –r 's/^([^:]*):([^:]*)/2:1/' Same as above. Exchanges only the first two
fn columns even if there are more.
sed –r 's/^(This (.*) nested)/2 Group 1 contains everything between "This ..
1/' fn nested". Group 2 contains just the characters
between "This" and "nested".
20. Substitution with Grouping and Back
Referencing. Examples
20
Command Explanation
sed –r Print lines with six or seven character long
'(w)(w)(w)w?321/p' fn palindromes
sed –r Convert six char palindrome strings to
's/(.)(.)(.)321/123123/g repetitive strings. Note: Any embedded or
' fn trailing six spaces also will match
sed -r ':a;s/(^|[^0-9.])([0- Add comma as thousands separator (Think
9]+)([0-9]{3})/12,3/g;ta' how easy it would have if we were to do this
numbers.txt from the left side)
sed –r 's/(.*)/11/' fn Concatenate the string at the line level
sed –r 's:(.*):11:' fn Same as above, but using : as the delimiter
between search and replacement strings
21. Inserting text with "i" and "a"
21
Example Meaning
sed '4i Inserts two lines before line 4
my text 1
my text 2' in_file
sed '/pat1/ i Inserts two lines before every line that contains
my text 1 pat1
my text 2' in_file
sed '4a Inserts two lines after line 4
my text 1
my text 2' in_file
sed '$a Inserts two lines after the end of the file
my text 1
my text 2' in_file
22. Changing text
22
Example Meaning
sed '4c Replace line 4 with the two new lines
my text 1
my text 2' in_file
sed '/pat1/ c Replace all lines containing pat1 with the two
my text 1 new lines
my text 2' in_file
sed '/pat1/, /pat2/ c Replace all lines between pat1 and pat2 with the
my text 1 two new lines
my text 2' in_file
Change affects the whole line.
Substitute just the matching words or strings on the line
23. Deleting lines
23
Command Result
sed '10d' in_file Delete the 10th line
sed '1,10d' in_file Delete lines between 1 and 10
sed '10,$d' in_file Delete lines from 10 to end of file
sed '10,+3d' in_file Delete line 10 and 3 lines below (lines 10, 11, 12 and 13)
sed '10~3d' in_file Delete every third line after line 10
sed '10, ~3d' in_file Delete line 10 and up to the next multiple of 3 (lines 10,11 and
12)
sed '/pat1/d' in_file Delete all lines containing pat1
sed '/pat1/,+3d' in_file Delete all lines containing pat1 and three lines following it
sed '/pat1/,~3d' in_file Delete lines containing pat1 and up to next the multiple of 3
sed '/pat1/!d' in_file Delete all lines NOT containing pat1
sed –r '/[0-9]{8,}/d' in_file Delete lines containing 8 or more digits
24. Printing lines
24
Command Result
sed –n '10,/pat1/p' in_file Print lines from 10 to next line containing pat1
sed –n '10,+3p' in_file Print line 10 and 3 lines below (lines 10, 11, 12 and 13)
sed –n '10~3p' in_file Print every third line after line 10
sed –n '10, ~3p' in_file Print line 10 and up to the next multiple of 3 (lines 10,11 and
12)
sed –n '/pat1/p' in_file Print all lines containing pat1
sed –n '/pat1/,+3p' in_file Print all lines containing pat1 and three lines following it
sed –n '/pat1/,~3p' in_file Print lines containing pat1 and up to next the multiple of 3
sed –n '/pat1/,/pat2/p' Repetitively print lines between the ranges containing
in_file pat1 and pat2
sed –n '/pat1/!p' in_file Print all lines NOT containing pat1
sed -n '/([0-9]{3})[0- Print lines containing phones numbers.
9]{3}-[0-9]{4}/p'
in_file
25. A note on the –r option
25
Command Result
sed -n '/([0-9]{3})[0- Print lines containing phones numbers like (408)806-
9]{3}-[0-9]{4}/p' 8330, (408)349-3699
in_file
sed -nr '/([0- Same as above with the –r option. Note the escaping of
9]{3})[0-9]{3}-[0- parentheses. Without escaping, ( and ) become meta
9]{4}/p' in_file characters, not part of the search string.
26. Printing with and without "-n"
option - Comparison
26
Command Comments
sed '1,10p' fn Lines 1 to 10 are printed twice. Rest of the lines are
printed once
sed –n '1,10p' fn Lines 1 to 10 are printed just once.
sed 's/pat1/pat2/p' fn Lines containing pat1 are printed twice after
substituting pat1 with pat2. Other lines are printed
once
sed –n 's/pat1/pat2/p' Only lines containing pat1 are printed after
fn substituting pat1 with pat2
27. Inserting/reading from a file
27
Command Result
sed '10r my_file' in_file Insert my_file after the 10th line
sed '1,10r my_file' in_file Insert my_file after each line between 1 and 10
sed '$r my_file' in_file Append my_file at the end
sed '10,+3r my_file' in_file Insert my_file after every line between10 and 13.
sed '10~3r my_file' in_file Insert my_file every third line after line 10
sed '/pat1/,/pat2/r my_file' Insert my_file after every line between pat1 and pat2
in_file
28. Writing selectively
28
Command Result
sed '1,10w nf' in_file Write lines 1 to 10 to new file "nf"
sed '/^pat1/w nf' in_file Write all lines beginning with pat1 to nf
sed '/pat1$/w nf' in_file Write lines that end with pat1
sed '/<pat1>/w nf' in_file Write lines that contain the WORD pat1 to nf
sed –e '1~2w odd_lines' Write odd numbered lines to odd_lines, and even
-e '2~2w even_lines` numbered files to even_lines
in_file
sed '/pat1/, /pat2/w nf' Write all lines between pat1 and pat2 to nf
in_file
sed –r '/[0-9]{3}-[0-9]{2}-[0- Write lines containing Social Security Number nnn-nn-nnnn to the
9]{4}w ssn' in_file file ssn. Note the use of –r option. Otherwise, { and } have to be
escaped.
sed –r '/(.)(.)(.)321/w Write lines containing palindromes to palin.txt
palin.txt' in_file
29. Transliterating (like ‘tr’)
29
Command Result
y/source/dest/ One to one character by character substitution.
source and dest strings have to be of the same
length
sed 'y/13579/aeiou/' fn replace 1 by a, 3 by e, and so on
30. Quitting from sed
30
Command Result
sed '15q' in_file Quit after the 15th line. Print 14 lines
sed '/<[Ee]nd>/q' in_file Quit after encountering the word End or end
31. Grouping multiple commands with { and }
31
sed –n '/pat1/,/pat2/ {
s/pat3/pat4/g
s/pat5/pat6/g
w new_file
p
}' inp_file
32. One liners
32
Command Result
sed 'p' in_file Duplicate all the lines one below the other
sed '/^$/ d' in_file Delete the blank lines from the input file
sed ‘s/./&:/80’ Add a colon after the 80th character. (replace the
eightieth occurrence of the pattern with itself and a
colon)
sed –r ‘s/^(.*) 1/1/' fn Replace duplicate strings
sed G in_file Add a blank line after everyline.
find . –type f –name Replace all occurrences of TableA with TableB in all
my_files.*.sql –exec sed –i my SQL scripts
‘s/TableA/TableB/g’ {} +
More oneliners http://www.catonmat.net/blog/sed-one-liners-
explained-part-one/
33. Additional concepts not covered
33
labels, branching to them
h, H: copy/append patternspace to holdspace
g, G: copy/append holdspace to patternspace
x: exchange the contents of the hold and pattern
spaces
34. References
34
sed & awk by Dale Dougherty & Arnold Robins
http://www.grymoire.com/Unix/Sed.html
http://sed.sourceforge.net/sedfaq3.html
http://en.wikipedia.org/wiki/Sed
http://groups.yahoo.com/group/sed-users/
http://sed.sourceforge.net/sed1line.txt
36. Unanswered questions
36
How to simulate tail with sed?
How to substitute the nth to mth occurrences of pat1
with pat2?
How to substitute the last N occurrences or the nth
occurrence from the end?
How to identify palindrome of any length?
Hinweis der Redaktion
In /pat1/,+3 if pat1 again appears within the next three lines, it will be ignored.
