Unix command line tricks
Original source: http://www.acck.edu/~jerod/spellbook.html
This page is always being modified.
This is a listing of short programs to accomplish small tasks. These were written for the csh on HP-UX 11.0
but should be applicable to any BSD derived system. You will find awk, sed, perl, and vi tricks here.
Frequently I work with files containing data that require preparation to use somewhere else.
A good way to get familiar is to create some test files filled with data. Here is a sample for you to
copy and paste in your text editor. For this page, this file will be called myfile.
Ashbourne 14567 Crowley, Tom crowley@fire
Belfast 65420 Stewart, Bob stewart@electric
Chelmsford 34591 Campbell, Lisa campbell@kinetic
Dungannon 11278 Morse, John morse@gravity
Edinburgh 2290 Fox, Paul fox@water
Fareham 34531 Biggs, Douglass biggs@metal
Glasgow 98231 Anderson, Aubrey anderson@air
Hastings 4500 Baldauf, Jack baldauf@snuke
Ipswich 32578 Brooks, David brooks@mineral
Jesmond 44498 Fryxell, Greta fryxell@wood
Kensington 90124 North, Gerald north@wnuke
London 55812 Pinckney, Jay pinckney@shadow
Manchester 98703 Slowey, Niall slowey@plasma
Newcastle 89239 Wormuth, John wormuth@earth
Display file without the blank lines:
awk 'NF > 0' myfile
awk '/./' myfile
sed -e '/^$/d' myfile
perl -ne 'print unless /^ *$/' myfile
You can delete the blank lines in vim by entering this command while the cursor is at the top of the file.
:g/^$/d
Delete blanks (whitespace) at start/end of line:
In vim, to remove leading blanks :%s/^ *//
In vim, to remove trailing blanks :%s/ *$//
This will not remove blanks between words.
awk '{sub(/^[ \t]+/, ""); print}' myfile # from start of line
awk '{sub(/[ \t]+$/, "");print}' myfile # from end of line
sed 's/\ \ *$//' myfile # remove blank space at end of line
Insert blanks at start of line.
perl -p -i -e "s/$/ /g;" myfile # insert blank space at end of line
perl -p -i -e "s/^/ /g;" myfile # insert blank space at start of line
Remove first 10 lines in a file:
sed '1,10d' myfile
Remove the last character in a line in vim:
:1,$s/.$//
Place a comma at the end of each line:
sed 's/$/\,/g' myfile
perl -p -i -e "s/\n/\,\n/g;" myfile
Insert blank lines in a file:
sed G myfile # single blank line between each line
sed "G;G" myfile # double blank lines between each line
sed "n;n;n;n;G;" myfile # insert blank line after 5 lines, 10 lines, etc.
awk '1;{print ""}' myfile # single blank line between each line
awk 'NF{print $0 "\n"}' # double blank lines between each line
awk '1;{print "\n"}' # triple blank lines
Place single quotes around line:
sed "s/\(.*\)/'\1'/" myfile
Enumerate non-blank lines at the beginning:
sed '/./=' myfile | sed '/./N; s/\n/ /'
Reverse the order of a file:
sed '1!G;h;$!d' myfile # this works in sh not csh
perl -e 'print reverse <>' myfile
in vim, enter this command: :g/^/m0
Left justify each line:
awk '{sub(/^[ \t]+/, ""); print}' myfile
Add/remove leading angle bracket at start of line:
sed 's/^/> /' myfile
sed 's/^> //' myfile
Print file without HTML tags (may not be 100% effective):
sed -e :a -e 's/<[^>]*>//g;/</N;//ba' myfile
Capitalize the first letter of each line:
awk '{ sub(/^./,toupper(substr($0,1,1))); print; }' myfile
Print even-numbered lines:
awk 'NR % 2' myfile
awk 'NR % 2 == 0' myfile # for odd-numbered lines
Take every other line and place in file, then take the remaining lines into second file
awk 'NR%2==1 {print >> "file1"; next} {print >> "file2"}' myfile
--- or ---
awk 'NR%2==0' myfile > file2
awk 'NR%2==1' myfile > file1
perl -ne 'print{$.%2?STDOUT:STDERR}$_' myfile 1> file1 2> file2
sed -ne 'w file1' -e 'n;w file2' myfile
Print certain lines:
sed -n '5,8p' myfile # print lines 5,6,7 and 8 of myfile
Beginning at line 2, print every 4th line:
sed -n '2,${p;n;n;n;}' myfile
Place colon between fields:
You need to know the number of characters in each line. In our example file myfile,
think of a number line running across the file:
---------1---------2---------3---------4---------5---------6
Ashbourne 14567 Crowley, Tom crowley@fire
Belfast 65420 Stewart, Bob stewart@electric
Chelmsford 34591 Campbell, Lisa campbell@kinetic
And determine the best character number to place a colon, then use the substr function of awk:
awk '{print substr($0,1,12),":",substr($0,12,7),":",substr($0,19,20),":",substr($0,39,20)}' myfile
will result in:
Ashbourne : 14567 : Crowley, Tom : crowley@fire
Belfast : 65420 : Stewart, Bob : stewart@electric
Chelmsford : 34591 : Campbell, Lisa : campbell@kinetic
Dungannon : 11278 : Morse, John : morse@gravity
Edinburgh : 2290 : Fox, Paul : fox@water
Fareham : 34531 : Biggs, Douglass : biggs@metal
Glasgow : 98231 : Anderson, Aubrey : anderson@air
Hastings : 4500 : Baldauf, Jack : baldauf@snuke
Ipswich : 32578 : Brooks, David : brooks@mineral
Jesmond : 44498 : Fryxell, Greta : fryxell@wood
Kensington : 90124 : North, Gerald : north@wnuke
London : 55812 : Pinckney, Jay : pinckney@shadow
Manchester : 98703 : Slowey, Niall : slowey@plasma
Newcastle : 89239 : Wormuth, John : wormuth@earth
Notice that awk automatically places a space flanking the colon.
Find & replace methods:
Change words to "ether"
sed "s/mineral/ether/" myfile # replaces 1st instance on each line
sed "s/water/ether/4" myfile # replaces 4th instance on each line
sed "s/metal/ether/g" myfile # replaces ALL instances
sed "/metal/s/metal/ether/g" myfile # same but executes more quickly
sed "/North/s/nuke/ether/g" myfile # ONLY for lines which contain "North"
sed "/North/!s/nuke/ether/g" myfile # EXCEPT for lines which contain "North"
perl -p -e 's/fire/ether/g;' myfile #show ether instead of fire
perl -p -i -e 's/fire/ether/g;' myfile #actually changes the contents of myfile
perl -pi -e'substr($_,6,3)="---"' myfile # change the 7th, 8th, and 9th characters of all lines to ---
awk '{sub(/gravity/,"ether");print}' myfile # replaces all gravity with ether
awk '/baz/{sub(/plasma/, "ether")};{print}' myfile # replaces plasma with ether only on lines with baz
awk '{sub(/fire|electric|kinetic/, "ether"); print}' myfile # change fire,electric,kinetic to ether
in vim:
:%s/shadow/light/g # replaces ALL instances of shadow with light in file
Change the 7th character of each line to the letter j
sed 's/./j/7' myfile
perl -pe '7 < length and substr($_,6,1)="j"' myfile
In the vim editor: :%s/^\(.\{7\}\)./\1j/
List directory files with a certain datestamp
ls -l | awk '{if($6=="May" && $7=="16") print $NF }'
Will list the names of files in the current directory dated May 16.
Find the file 'myfile' in subdirectories, starting with the current
directory, but don't bother looking in the 'docs' subdirectory
find . ! -path ./docs -only -name myfile # yes, I am aware that this is not using vi, perl, sed or awk
find . \( -name 'myfile' -o \( -name 'docs' -prune \) \) -type f -print
Get the lines flanking lines with search term
awk '/Manchester/{print x};{x=$0}' myfile; awk '/Manchester/{getline;print}' myfile
will get you the lines before and after lines with 'Manchester':
London 55812 Pinckney, Jay pinckney@shadow
Newcastle 89239 Wormuth, John wormuth@earth
Print the binary equivalent of the word 'fire'
perl -e 'print unpack "B*","fire"' ; perl -e 'print "\n"'
Print the hex equivalent of the word 'fire'
perl -e 'print unpack "H*","fire"' ; perl -e 'print "\n"'
To reverse the translated string back, use pack instead of unpack
Back