Unix shell scripting

vybhavatechnologies.blogspot.in
UNIX
&
LINUX
Shell Scripting

Course
Outline
What | Who | How

Course objective
• Know about UNIX, shell, types of shells
• Identifying the shell that suites to your needs
• Users and groups in a shell, permissions
• Use archiving and compression commands
• Use basic regular expressions
• Identify process management and job control
• Identify the Linux Filesystem
• Resource management
• Network resources, remote communications
• Use text editors like vi, vim
• Write the automation scripts

Scope : Who can do this?
• System Administrators
– Beginner Linux Admin
– DBA
– WebLogic Admin
– WebSphere Admin
– Jboss Admin
• Application Developers
• Configuration & Release managers (SCM Practice)
• Support team members
• Technical lead, Managers
• Automation Specialist
• Web Administrators
• Security team

Overview of
UNIX & Linux
UNIX, History, influencers

Overview of UNIX & Linux
• UNIX Operating System
• UNIX Evolution
• UNIX and Linux Influencers
• Flavors Of UNIX
• Why UNIX?
• Unix and Linux Features

UNIX Operating System
• Unix is a layered operating system.
The innermost layer is the hardware that provides the services for the OS.
• The operating system, referred to in Unix as the kernel, interacts directly
with the hardware and provides the services to the user programs.
• User programs interact with the kernel through a set of standard system
calls.

UNIX History

UNIX and Linux Influencers
Linus
Torvalds
Richard Stallman
Dennis Riche
Ken Thompson

Flavors of UNIX

Trend : Virtual Boxes
• Virtual Box

Why UNIX?
• PC magazine report

UNIX and Linux Features
• Its first real use is as a text processing tool for the patent
department at Bell Labs. It was designed with these features:
• Programmers environment
• Simple user interface
• Simple utilities that can be combined to perform powerful functions
• 1973 Unix is re-written mostly in C, a new language developed by Dennis
Ritchie. Being written in this high-level language greatly decreased the effort
needed to port it to new machines.
• And transparent to the user.

vybhavatechnologies.blogspot.com
Directory structure

CUI Editors
Text editors
emacs
GUI Editors

The vi editor

Kernel
• Core of the Unix system.
• Interacts directly with the OS
• Largely written in C, with some parts written
in Assembly language.
• Insulates other parts of OS from hardware.
• Performs all low level functions.
• All the applications deal with Kernel.
• Handles all the devices.

Kernel Functions
• Memory management.
• Process scheduling.
• File management and security.
• Interrupt handling and error reporting.
• Input / Output services.
• System accounting

Shell
• Acts as an interface between the user and the
kernel.
• Is the command interpreter of UNIX.
• Provides powerful programming capabilities.
• System administration.
• Text processing.
• File management.
• UNIX Utilities and Applications.
• User can customize his/her own shell.
• Users can use different shells on the same
machine.

Book References

Online shells
• Linux zoo : linuxzoo.net
• There is 15 min shell : http://simpleshell.com
• Short time shell : http://cb.vu/
• Side by side shell & executed outputs :
http://www.compileonline.com/execute_bash_online.php
• Cloud Shells:
– Amazon EC2
– Ibm cloud instance

Shell Basics
SHELL hierarchy, types, Login
shell

UNIX Shell basics
• Shell hierarchy
• Types of Shells
• Know about Shell
• Login Shell
• Why Use Shell Script?

Types of shells

Shell Features
• Filename completion
• Command-line history.
• Multiple job control.
• Redirection (program to file).
• Pipes (program to program).
• Storage variables.
Some of the common features that most all shells have to offer:

Booting Process
• Hardware configuration check.
• The operating system file /vmunix ( the UNIX
kernel) gets loaded.
• The kernel brings up the swapper and the init
program.
• Jobs of init are:
– Takes the system into the multi-user mode.
– Checks the integrity of the file systems.
– Mounts the file systems
– The login process gets executed.
– After successful login, the shell appears.

Run Levels
• UNIX system has several modes of operation
called system states ( or run levels). These are:
0 : shutdown state
1 : administrative state
s or S : single user state
2 : multi-user state
6 : stop and reboot state
• These states are passed to the init program,
which operates the system in the appropriate
state.

Login shell

I am in which shell?
• The current shell program that is in use is stored in environment variable
SHELL.
• To find the current shell program, use following command
• $ echo $SHELL
• /bin/bash
• $

Why Use Shell scripts ?
• Batch processing of commands
• Repetitive tasks
• You can use shell scripts to automate administrative tasks.
• Encapsulate complex configuration details.
• Get at the full power of the operating system.
• The ability to combine commands allows you to create new commands
Adding value to your operating system.

Shell
command-line
Variables, Math, Commands

Shell command line
• Variables and strings
• Math in Scripting
• Shell Commands

Variable and Strings
• Variables prefix with $ is a variable.
– $VAR ${VAR}
• Examples: ${FILE}.txt
• echo “my name is $NAME”
• echo “my name is “$NAME
• read VAR
• read VAR1 VAR2
• Strings are enclosed in “ ” or just written with spaces escaped by VAR=”How
are you”
• VAR=How are you
• VAR=HOW are you

Math in scripting
• Use $((…)) instead of expr to evaluate arithmetic equation
• x=1
• x=$((x+1))
• echo $((1+1))
• echo $[1+1]
• echo `expr 1 + 1`
• A=$((B*C))
• let A+=1
• echo "3.4+7/8-(5.94*3.14)" | bc bc
let
$(( ))
expr

An aside: $(( )) for Math
% echo $(( 1 + 2 ))
3
% echo $(( 2 * 3 ))
6
% echo $(( 1 / 3 ))
0

Math with bc
• bc ("basic calculator?")
bc is a handy but full-featured calculator. Even includes a programming language!
Use the -l option to have floating point support.
• z=10; z=`echo "$z + 1" | bc`

Shell commands
• date: Displaying the System Date
• cal: The Calendar
• echo: Displaying a Message
• printf: An Alternative echo
• script: Recording Your Session
• Email Basics: mailx: The Universal Mailer
• passwd: Changing Your Password
• who: Who are the users?
• uname: Knowing Your Machine's Characteristics
• tty: Knowing Your Terminal
• stty: Displaying and Setting Terminal Characteristics

The date command
• The date command returns the current date or time.
• Useful in many scripts to record when commands started or completed.
Example:
% DATESTRING=`date +%Y%m%d`
% echo $DATESTRING
20060125
% man date

Customizing
ENV
path, display, expor
t

Customizing Environment
• How to Setup Environment?
• Environment variables
• The history command options
• Shortcuts : Defining alias
• Utility scripting
• Scoping with export

How to Setup Environment?
• The Common Environment Variables
• Aliases (bash and ksh)
• Command History (bash and ksh)
• In-line Command Editing (bash and ksh)
• To override the value of an environment variable for a single command, just
specify the desired value before the command: ENV[„PATH‟]
• The Initialization Scripts

Environment variables
• $HOME home directory
• $PATH path
• $PS1 user prompt (normally %)
• $PS2 secondary prompt (normally >)
• $$ process id of the script
• $# number of input parameters
• $0 name of the script file
• $IFS separation character (white space)
• Use „env‟ to check the value

The history commands
• The history command will show you a list of the most recently used
commands, with command sequence numbers. To specify a previously
issued command:
• use the up and down arrow keys to scroll through the history
• use ”!” and the command number, or “!!” for the most recent command.
• use [ctrl-r]: recall a previous command containing a desired string by typing
[ctrl-r] and the string. Then continue to press [ctrl-r] to cycle through the
matching commands. They can be edited.

Shortcuts :Defining alias
• Global alias can be defined in .profile file you can test in command line as well
– Syntax: alias name=value
– Here the value can be any command or script paths
• Examples:
– alias -g L="|less"
– alias c='clear„
– alias now='date +"%T'
– alias -g ...='../..'
– alias ping='ping -c 5„
– alias update='yum update„
– alias meminfo='free -m -l -t'
– alias apt-get='sudo apt-get„
• You can do unalias also
• unalias aliasname
• Example :
– unalias L

Utility scripts
• Frequently used tasks can be turn to utility

Scoping with export
• Without export
• TUTOR=„Shekhar‟
• bash
• echo $TUTOR
• With export

Shell
Programming
Execution

Shell Programming
• Shell Programming
• Program structure
• Shell variables
• Quotes - Strings
• Shell Output
• Print formatted data
• Continuing Lines
• Exit Status
• Logic using test
• Expressions

How shell works?
• Execution cycle for „ls‟ command

Shell defaults
• $# : number of arguments on the command line
• $- options supplied to the
• ? exit value of the last command executed
• $n argument on the command line, where n is from 1 through
9, reading left to right
• $0 the name of the current shell or program
• $* all arguments on the command line ("$1 $2 ... $9")
• $@ all arguments on the command line, each separately
quoted ("$1" "$2" ... "$9")
• $argv[n] selects the nth word from the input list
• $#argv report the number of words in the input list
• $$ process number of the current process
• $! process number of the last command done in background

Shell programming
• We can write a script containing many shell commands
• Interactive Program:
– grep files with POSIX string and print it
% for file in *
> do
> if grep –l POSIX $file
> then
> more $file
 fi
 done
Posix
There is a file with POSIX in it
– „*‟ is wildcard
% more `grep –l POSIX *`
% more $(grep –l POSIX *)
% more –l POSIX * | more

Shell variables
• Variables needed to be declared,
• Note it is case-sensitive (e.g. foo, FOO, Foo are different values)
• Add „$‟ for storing values
% salutation=Hello
% echo $salutation
Hello
% salutation=7+5
% echo $salutation
7+5
% salutation=“yes dear”
% echo $salutation
yes dear
% read salutation
Hola!
% echo $salutation
Hola!

Quotes - Strings
• Edit a “vartest.sh” file
#!/bin/sh
clear
myvar=“Hi there”
echo $myvar
echo “$myvar”
echo `$myvar`
echo $myvar
echo Enter some text
read myvar
echo „$myvar‟ now equals $myvar
exit 0
Output
Hi there
Hi there
$myvar
$myvar
Enter some text
Hello Scripting
$myvar now equals Hello Scripting

Shell Output - echo
• echo command used to print string and variable values
• -n do not output the trailing newline
• -e enable interpretation of backslash escapes
– 0NNN the character whose ACSII code is NNN
– backslash
– a alert
– b backspace
– c suppress trailing newline
– f form feed
– n newline
– r carriage return
– t horizontal tab
– v vertical tab
Try following examples
% echo –n “string to n output”
% echo –e “string to n output”

Print formatted data
• printfformat and print data
• Escape sequence
– backslash
– a beep sound
– b backspace
– f form feed
– n newline
– r carriage return
– t tab
– v vertical tab
• Conversion specifies
– %d decimal
– %c character
– %s string
– %% print %
Example:
% printf “%sn” hello
Hello
% x=100
% printf “%s %dt%s” “Hi There”,
$x people
Hi There 100 people

An aside: Quoting
% echo „$USER‟
$USER
% echo “$USER”
pavanusr
% echo “””
”
% echo “vybhavatrainings”
Vybhavatrainings
% echo „”‟
”

Continuing Lines:
% echo This
Is
A
Very
Long
Command Line
This Is A Very Long Command Line
%

Exit Status
• $?
• 0 is True
% ls /does/not/exist
% echo $?
1
% echo $?
0

Exit Status: exit
• exit nending the script
• 0 means success
• 1 to 255 means specific error code
• 126 means not executable file
• 127 means no such command
• 128 or >128 signal
% cat > test.sh <<END
exit 3
END
% chmod +x test.sh
% ./test.sh
% echo $?
3

Logic: test
• test or „ [ ] „
if test –f fred.c
then
...
Fi
% test 1 -lt 10
% echo $?
0
% test 1 == 10
% echo $?
1
If [ -f fred.c ]
then
...
fi
-d file if directory
-e file if exist
-f file if file
-g file if set-group-id
-r file if readable
-s file if size >0
-u file if set-user-id
-w file if writable
-x file if executable
need space !

Logic: test
• test
• [ ]
– [ 1 –lt 10 ]
• [[ ]]
– [[ “this string” =~ “this” ]]
• (( ))
– (( 1 < 10 ))

Expressions
• Conditional operations
– expression1 –eq expression2
– expression1 –ne expression2
– expression1 –gt expression2
– expression1 –ge expression2
– expression1 -lt expression2
– expression1 –le expression2
– !expression
• String validation
– String1 = string2
– String1 != string 2
– -n string (if not empty string)
– -z string (if empty string)
• [ -f /etc/passwd ]
• [ ! –f /etc/passwd ]
• [ -f /etc/passwd –a –f /etc/shadow ]
• [ -f /etc/passwd –o –f /etc/shadow ]

Shell Script
Controls
If, case, while, for
loop

Shell script controls
• Shell script if control
• Switching with case
• while: Looping
• for: Looping with a List
• break and continue in loops
• trap: Interrupting a program
• Debugging Shell Scripts
• Exercise

If-else Control
Syntax
if condition
then
statement
else
statement
fi
66
#!/bin/sh
echo “Is it morning? Please answer yes or no”
read timeofday
if [ $timeofday = “yes” ]; then
echo “Good morning”
else
echo “Good afternoon”
fi
exit 0
Is it morning? Please answer yes or no
yes
Good morning

If-elif-else control
67
#!/bin/sh
read timeofday
if [ $timeofday = “yes” ]; then
elif [ $timeofday = “no” ]; then
else
echo “Sorry, $timeofday not recognized. Enter yes or no”
exit 1
fi
exit 0

If with exit
68
#!/bin/sh
read timeofday
if [ “$timeofday” = “yes” ]; then
elif [ $timeofday = “no” ]; then
else
echo “Sorry, $timeofday not recongnized. Enter yes or no”
exit 1
fi
exit 0
If input “enter” still returns Good morning

If and case Controls
• If then, else, elif, fi
1. if [ -d $F ]; then
2. rm -rdf $F
3. elif [ -f $F ]; then
4. rm $F
5. else
6. echo “Unknown file type”
7. fi
• case, esac
– while read l; do
– case $l in 1) echo "One";;
– 2) echo "Two";;
– 3) echo "Three";;
– *) echo "Invalid"; break;;
– esac done

Case Statement
Syntax
case variable in
pattern [ | pattern ] …) statement;;
pattern [ | pattern ] …) statement;;
…
esac
70
#!/bin/sh
read timeofday
case “$timeofday” in
yes) echo “Good Morning”;;
y) echo “Good Morning”;;
no) echo “Good Afternoon”;;
n) echo “Good Afternoon”;;
* ) echo “Sorry, answer not recongnized”;;
esac
exit 0

Case Statement
• A much “cleaner” version
71
#!/bin/sh
read timeofday
yes | y | Yes | YES ) echo “Good Morning”;;
n* | N* ) echo “Good Afternoon”;;
* ) echo “Sorry, answer not recongnized”;;
esac
exit 0
But this has a problem, if we enter „never‟ which obeys n*
case and prints “Good Afternoon”

Improved Case Statement
72
#!/bin/sh
read timeofday
yes | y | Yes | YES )
echo “Good Morning”
echo “Up bright and early this morning”
;;
[nN]*)
echo “Good Afternoon”;;
*)
echo “Sorry, answer not recongnized”
echo “Please answer yes of no”
exit 1
;;
esac
exit 0

Loops in shell scripting
• While loop
• while read VAR; do
• echo $VAR;
• done
• For each loop
• for VAR in {1..5}; do
• echo $VAR;
• done
• for VAR in 1 2 3 4 5; do
• echo $VAR;
• done
• for VAR in {0..10..2}; do
• echo $VAR; done
• for VAR in $(seq 0 2 10); do
• echo $VAR;
• done
• For loop with counter
• for (( i=0; i<=10; i+=2 )); do echo
$i;
• done

Control break
• Internal:
• only in script
• break
74
#!/bin/sh
rm –rf fred*
echo > fred1
echo > fred2
mkdir fred3
echo > fred4
for file in fred*
do
if [ -d “$file” ] ; then
break;
fi
done
echo first directory starting fred was $file
rm –rf fred*
exit 0

Continue loops
• Continue Continues next iteration
75
#!/bin/sh
rm –rf fred*
echo > fred1
echo > fred2
mkdir fred3
echo > fred4
for file in fred*
do
if [ -d “$skipdir” ]; then
echo “skipping directory $skipdir”
continue
fi
echo file is $file
done
rm –rf fred*
exit 0

The trap Command
• trap action after receiving signal
trap command signal
• signal explain
HUP (1) hung up
INT (2) interrupt (Crtl + C)
QUIT (3) Quit (Crtl + )
ABRT (6) Abort
ALRM (14) Alarm
TERM (15) Terminate
76

Example of trap Command
77
#!/bin/sh
trap „rm –f /tmp/my_tmp_file_$$‟ INT
echo creating file /tmp/my_tmp_file_$$
date > /tmp/my_tmp_file_$$
echo “press interrupt (CTRL-C) to interrupt …”
while [ -f /tmp/my_tmp_file_$$ ]; do
echo File exists
sleep 1
done
echo The file no longer exists
trap INT
echo creating file /tmp/my_tmp_file_$$
date > /tmp/my_tmp_file_$$
echo “press interrupt (CTRL-C) to interrupt …”
while [ -f /tmp/my_tmp_file_$$ ]; do
echo File exists
sleep 1
done
echo we never get there
exit 0

Debug scripts
• sh –n<script> set -o noexec check syntax
set –n
• sh –v<script> set -o verbose echo command before
set –v
• sh –x<script> set –o trace echo command after
set –x
set –o nounset gives error if undefined
set –x
set –o xtrace
set +o xtrace
trap „echo Exiting: critical variable =$critical_variable‟ EXIT
78

Exercise
• Create script that prints 1-10 all even numbers
• Create script that prints 3 multiples skip printing 9 and 18 terminate at 30.
• Print prime numbers to given range.
• Print Fibonacci numbers given range.

Shell &
Sub-shells
Functions, fork, exec

Sub-shells
• Function
• Shell and sub-shell
• The process system calls fork, exec, exit, wait
• Array in shell

Function
• You can define functions for “structured” scripts
function_name() {
statements
}
82
#!/bin/sh
foo() {
echo “Function foo is executing”
}
echo “script starting”
foo
echo “script ended”
exit 0
Output
script starting
Function foo is executing
Script ended
You need to define a function before using it
The parameters $*,$@,$#,$1,$2 are replaced by local value
if function is called and return to previous after function is finished

Defining a Function
83
#!/bin/sh
sample_text=“global variable”
foo() {
local sample_text=“local variable”
echo “Function foo is executing”
echo $sample_text
}
echo “script starting”
echo $sample_text
foo
echo “script ended”
echo $sample_text
exit 0
Output?
Check the
scope of the
variables
define local
variable

The function returns
• Use return to pass a result
84
#!/bin/sh
yes_or_no() {
echo “Is your name $* ?”
while true
do
echo –n “Enter yes or no:”
read x
case “$x” in
y | yes ) return 0;;
n | no ) return 1;;
* ) echo “Answer yes or no”
esac
done
}
echo “Original parameters are $*”
if yes_or_no “$1”
then
echo “Hi $1, nice name”
else
echo “Never mind”
fi
exit 0 Output
./my_name John Chuang
Original parameters are John Chuang
Is your name John?
Enter yes or no: yes
Hi John, nice name.

Command execution
• $(command) to execute command in a script
• Old format used “`” but it can be confused with “‟”
#!/bin/sh
echo The current directory is $PWD
echo the current users are $(who)

Shells and Sub-shells
• () and {}: Sub-shell 0r Current Shell?
• export: Exporting Shell Variables
• Running a Script in the Current Shell: The . Command
• let: Computation-a Second Look (Korn and Bash)
• Arrays (Korn and Bash)
• String Handling (Korn and Bash)
• Conditional Parameter Substitution
• Merging Streams
• Shell Functions
• eval: Evaluating Twice
• The exec Statement

Fork, exec, exit, wait

Array in Shell
• Declaring Array
• declare -a Linux=('Debian' 'Red hat' 'Suse' 'Fedora');
• Printing Array
• echo ${Linux[@]}
• Debian Red hat Ubuntu Suse
• Array Size
• echo ${#Linux[@]} #Number of elements in the array
• 4
• echo ${#Linux} #Number of characters in the first element of the array.i.e
Debian
• 6

File
System
NAVIGATION, FILE, DIR,
PERMISSIONS

File commands
• What's in a (File) name?
• The Parent-Child Relationship
– The HOME Variable: The Home Directory
– Current Directory
– cd: Changing the Current Directory
• mkdir: Making Directories
• rmdir: Removing Directories
• Absolute Pathnames
• Relative Pathnames
• ls: Listing Directory Contents
• The UNIX File System

Everything is a file
Regular files Almost everything in Unix is a file!
Directories
Directory are just files listing a set of files
Symbolic links
Files referring to the name of another file
Devices and peripherals
Read and write from devices as with regular files
Pipes
Used to cascade programs cat *.log | grep error
Sockets
Inter process communication

What is file?
• In UNIX everything is a file
• Files can be tested with file command with filename.
• Text file
• Executable file
• ASCII file
• Binary file

File system hierarchy
• Root and directories
/
bin etc dev usr home
passwd
passwd hosts
local lib

Change directory
• > cd /etc # change to absolute path
• > cd mypath # change to relative path
• > cd ~ # change to home directory
• > cd # same as cd ~ (unlike DOS/Windows)
• > cd - # toggles current and previous directory
• > pushd/popd # directory stack
• If the shell finds a CDPATH environment variable, then all
directories in that path will be searched for the subdirectory when a relative cd
is executed:
• > export CDPATH=~/docs
• > cd a_doc_subdir # will work even when not in ~/docs

Make directory
• Creates subdirectories with all necessary intermediate directories in a single
command:
> mkdir -p a/really/deep/directory/tree
• A shortcut for capturing the last parameter in the previously executed
command is !$, so you can do this to change to that new directory:
> cd !$
• As stated before, to jump back up, you can do:
> cd -

Remove directory
• Remove a file with the rm, remove,
• rm [options] filename
• -i interactive (prompt and wait for confirmation before proceeding)
• -r recursively remove a directory, first removing the files and subdirectories
beneath it
• -f don't prompt for confirmation (overrides -i)

File system basics
• / is not allowed in a file name
• file extensions are not important
• File and folder names are case sensitive
• Names containing spaces are escaped by or ""
• Hidden files start with . (dot)
• Current directory alias . and parent directory alias ..
• ls, ls -a, ls -l, cp, mv, rm, rm -rdf, ln, ln -sf

The ls command
• Lists the files in the current directory, in alphanumeric order,except files
starting with the “.” character.
• ls -a (all) Lists all the files (including .* files)
• ls -l (long) Long listing (type, date, size, owner, permissions)
• ls -t (time) Lists the most recent files first
• ls -s (size) Lists the biggest files first
• ls -r (reverse) Reverses the sort order
• ls -ltr (options can be combined) Long listing, most recent files at the end

File name pattern
• Better introduced by examples!
• ls *txt
The shell first replaces *txt by all the file and directory names ending by txt
(including .txt), except those starting with ., and then executes the ls
command line.
• ls -d .*
Lists all the files and directories starting with .
-d tells ls not to display the contents of directories.
• cat ?.log
Displays all the files which names start by 1 character and end by .log

File handling with shell
• The File
• What's in a (File)name?
• The Parent-Child Relationship
• The HOME Variable: The Home Directory
• Absolute Pathnames
• Relative Pathnames
• ls: Listing Directory Contents
• The UNIX File System

File maintenance : cp, mv
• Copy the contents of one file to another with the cp command.
• cp [options] old_filename new_filename
• Common Options
• -i interactive (prompt and wait for confirmation before proceeding)
• -r recursively copy a directory
• Rename a file with the move command, mv. mv [options] old_filename
new_filename

Output file redirection
• Output redirection takes the output of a command and places it into a
named file. Input redirection reads the file as input to the command.
• > - output redirect
• >> append output

Input file redirection
• < input redirection
• <<String read from standard input until "String" is encountered as the only
thing on the line.

The cat options
• Creating Text Files with cat The “cat” command is used to create small text
files without using an editor.
• The cat utility reads the input from the user and writes it to its output. The
Ctrl-D (end-of-file, or EOF) key combination is used to tell cat that there is
no additional input.
• Display the contents of a file with the concatenate command
• cat [options] [file]
• -n precede each line with a line number
• -v display non-printing characters, except tabs, new-lines, and form-feeds
• -e display $ at the end of each line
• more, less, and pg - page through a file
• more, less, and pg let you page through the contents of a file one screenful
at a time

File display options
• more [options] [+/pattern] [filename]
• less [options] [+/pattern] [filename]
• pg [options] [+/pattern] [filename]
• head - display the start of a file
• head displays the head, or start, of the file.
• head [options] file
• -n number number of lines to display, counting from the top of the file
• tail - display the end of a file
• tail displays the tail, or end, of the file. tail [options] file
• -number number of lines to display, counting from the bottom of the file

The head command
Command by default displays first 10 lines of the
file.
The general command line format is:
$ head [-n] <file> <Enter>
Example:
head ae_socket.c  Will display first 10 lines
head –5 ae_socket.c  Will display first 5 lines

The tail command
• tail Command by default displays last 10 lines of the
file.
$ tail [+/-n] <file> <Enter>
Example:
tail ae_socket.c  Will display last 10 lines
tail –5 ae_socket.c  Will display last 5 lines
tail +5 ae_socket.c  Will display lines starting from
5th line
till end of file

Compare files
• cmp: Comparing Two Files
• comm: What is Common?
• diff: The diff command outputs the difference, if any, between two text files.
> diff before.txt after.txt

Compare and Contrast
• comm
Display common lines in two files.
Produces a three-column output:
Column 1: Lines that appear only in file1,
Column 2: Lines that appear only in file2,
Column 3: Lines that appear in both files.
$ comm [-[1] [2] [3]] file1 file2

• cmp
• Compare two files.
• No comment if the files are the same.
• If they differ, it announces the byte and line number at which the
difference occurred.
$ cmp file1 file2

• diff:
• Compare directories:
–diff sorts the contents of the directories by name.
–then runs the regular file diff algorithm on text files that
have the same name in each directory but are different.
• Compare file:
–diff tells what lines must be changed in the files to bring
them into agreement.
–Option:
• -b: ignores trailing blanks (spaces and tabs) and other strings.
• -i: ignore uppercase and lower case differences.
• -w: ignore all white spaces.

Compression archive files
• gzip and gunzip: Compressing and Decompressing Files
• tar: The Archival Program
• zip and unzip: Compressing and Archiving Together
• In addition to tar, the zip and unzip command line tool scan be very
handy. To see a zip file's content:
> unzip -v myzipfile.zip
• Combine with grep to find files whose names end in “html”:
> unzip -v myzipfile.zip | grep html$
• To create a zip file in a backup directory, containing the current
directory and all its subdirectories, do:
>zip -r ~/bu/docs-YYYYMMDD-HHMM.zip *

File system in UNIX
• Linux supports the following file systems:
minix, ext, ext2, xia, msdos, umsdos, vfat, proc, nfs, iso9660, hpfs, sysv, sm
b, ncpfs
• man fs
• File name no longer than 256 char
• Single path name should not exceed 1023
• mount /dev/hda/usr
• unmount
• File types:
• Binary
– Gif,jpeg, png
• Text
– Shell scripts, documents

File
Security
Chmod, chown, chgrp, umask

File Permissions
4bits type r w xr w x r w xu | g | s
owner group others
sticky
sgid
suid
4 2 1 4 2 1 4 2 1
- file
d directory
s local socket
l symbolic link
b block special file
c char special file
p named piped file

Changing Permissions
• chmod
• chown
• chgrp
chmod:
• Used to change the permissions of files and directory.
• Only the owner or super user can use chmod to change the
permissions of a file.
For example: to remove read write and execute permissions
on the file ae_socket.c for the group and others, type:
$ chmod go-rwx ae_socket.c

Options with chmod..
Symbol Meaning
u user
g group
o other
a all
r read
w write (and delete)
x execute (and access directory)
+ add permission
- take away permission

Ownership and Permissions
• Group and Ownership
• chown, chgrp
– chown user:group file
• sudo chown
• Permissions of files and directories can be added with + and revoked with –
with access specifier
• chmod 755 file
• chmod +x file
• chmod -x file
• (r)ead (w)rite e(x)ecute 4 2 1

Using numbers
• 1 - Execute
• 2 - Write
• 4 – Read
Example:
Original permission for ae_socket.c:
-rw----rw-
Thus in order to give read write permission to owner, group and
only read permission to others we would give:
Using character modes:
chmod o-w 1.c
chmod g+rw 1.c
Using numbers:
chmod 664 ae_socket.c

Common Permissions with Unmask
• Applying umask to file

The chmod command
• chmod -R a+rX linux/
Makes linux directory content and everything in it available to everyone!
• R: apply changes recursively
• X: x, but only for directories and files already executable
Very useful to open recursive access to directories, without adding
execution rights to all files.

Chmod contd…
• chmod a+t /tmp
• t: (sticky). Special permission for directories, allowing only the directory and
file owner to delete a file in a directory.
• Useful for directories with write access to anyone,
like /tmp.
• Displayed by ls -l with a t character.

Groups and users
• Users, su, sudo
• su, sudo, visudo, /etc/sudoers, su user
• sudo bash
• /etc/passwd

Symbolic link
• Find yourself often navigating several levels deep, many times? Create a
symbolic link:
ln -s work/yadameter/biz/bbsinc/yadameter y
• When you don‟t need it anymore, just remove the link with rm command:
rm y
• The directory pointed to will remain untouched; only the link will be deleted.
i Interactive
s Symbolic
f forcefully

• All links to a file access the same physical file.
• Modifying contents from any access changes the
file.
• Links can be of two type:
– Hard link.
• Created using ln <old file> <link name>
• Cannot create a link to directory.
• Can not span file systems.
– Soft Link.
• Created using ln –s <old file> <link name>
• Can also create link to directory.
• Can span file systems.

File
Filters
Pr, more, less, tail, head, sort,
uniq

File filters
• The Sample Database
• pr: Paginating Files
• head: Displaying the Beginning of a File
• tail: Displaying the End of a File
• cut: Slitting a File Vertically
• paste: Pasting Files
• sort: Ordering a File
• uniq: Locate Repeated and Nonrepeated Lines
• tr: Translating characters
• An example: Displaying a Word-count List

The cut filter
• select parts of a line
• The cut command allows a portion of a file to be extracted for another use.
• cut [options] file
• -c character_list character positions to select (first character
• -d delimiter field delimiter (defaults to <TAB>)
• -f field_list fields to select (first field is 1
• Both the character and field lists may contain comma-separated or blank-
character-separated numbers (in increasing order), and may contain a
hyphen (-) to indicate a range. Any numbers missing at either before (e.g. -
5) or after (e.g. 5-)
• the hyphen indicates the full range starting with the first, or ending with the
last character or field, respectively. Blank-character-separated lists must be
enclosed in quotes. The field delimiter should be enclosed in quotes
• is 1

cut - Example
To display the employee code, firstname and basic salary
from the employee file created earlier.
$ cut -d" " -f2,3,8 emp_new.txt
Where:
-d is used to specify the field delimiter. A space in this case.
-f represents fields and the numbers after it are the field
positions.
The specified columns are copied to the standard output.

cut - Examples
• To display characters 4 to 9 both inclusive, from
each record of the file employee:
$ cut -c4-9 emp_new.txt
• To display from the second to the fourth fields:
$ cut -d" " -f2-4 emp_new.txt
• To display from the fourth field onwards:
$ cut -d" " -f4- emp_new.txt

The paste filter
• paste - merge files
• The paste command allows two files to be combined side-by-side. The
default delimiter between the columns in a paste is a tab, but options allow
other delimiters to be used. Syntax
• paste [options] file1 file2
• Common Options
• -d list of delimiting characters
• -s concatenate lines

tr - translate characters
• The tr command translates characters from stdin to stdout. Syntax
• tr [options] string1 [string2]
• With no options the characters in string1 are translated into the characters
in string2, character by character in the string arrays. The first character in
string1 is translated into the first character in string2, etc.
• A range of characters in a string is specified with a hyphen between the
upper and lower characters of the range, e.g. to specify all lower case
alphabetic characters use '[a-z]'.

Sort file contents
• The sort command is used to order the lines of a file. Without any options,
the sort compares entire lines in the file and outputs them in ASCII order
• sort [options] [+pos1 [ -pos2 ]] file
– -k keydef sort on the defined keys (not available on all systems)
– -i ignore non-printable characters
– -n numeric sort
– -b ignore leading blanks (<space> & <tab>) when determining starting
and ending characters for the sort key
– -d dictionary order, only letters, digits, <space> and <tab> are significant
– -f fold upper case to lower case
– -o outfile output file
– -r reverse the sort
– -t char use char as the field separator character
– -u unique omit multiple copies of the same line

The tee filter
• tee - copy command output
• tee sends standard in to specified files and also to standard out. It's often
used in command pipelines. tee [options] [file[s]]
• -a append the output to the files
• -i ignore interrupts

The uniq command
• uniq filters duplicate adjacent lines
from a file. Syntax
• uniq [options] [+|-n] file [file.new]
• -d one copy of only the repeated lines
• -u select only the lines not repeated
• +n ignore the first n characters
• -s n same as above (SVR4 only)
• -n skip the first n fields, including any
blanks

Word count wc
• wc - count words in a file
• wc stands for "word count"; the command can be used to count the number
of lines, characters, or words in a file. Syntax
• Common Options
• -c count bytes
• -m count characters (SVR4)
• -l count lines
• -w count words

Regular
Expressions
Pattern matching, text filters,
grep, find, awk, sed

Regular Expressions
• Regular Expression Overview
• RE Character Classes
• RE Quantifiers
• RE Parenthesis
• RE Examples with grep
• Re in awk
• RE in sed

Regex overview
• A regular expression (abbreviated as regexp or regex, with plural forms
regexps, regexes, or regexen) is a string that describes or matches a set
of strings, according to certain syntax rules.
• Syntax
– ^ Matches the start of the line
– $ Matches the end of the line
– . Matches any single character
– [] Matches a single character that is contained within the brackets
– [^] Matches a single character that is not contained within the brackets
– () Defines a "marked subexpression”
– {x,y}Match the last "block" at least x and not more than y times

How regex works?
• Examples of regular expressions:
– ".at" matches any three-character string like hat, cat or bat
– "[hc]at" matches hat and cat
– "[^b]at" matches all the matched strings from the regex ".at" except bat
– "^[hc]at" matches hat and cat but only at the beginning of a line
– "[hc]at$" matches hat and cat but only at the end of a line

Pattern match class
• POSIX class similar to meaning
• [:upper:] [A-Z] uppercase letters
• [:lower:] [a-z] lowercase letters
• [:alpha:] [A-Za-z] upper- and lowercase letters
• [:alnum:] [A-Za-z0-9] digits, upper- and lowercase letters
• [:digit:] [0-9] digits
• [:xdigit:] [0-9A-Fa-f] hexadecimal digits
• [:punct:] [.,!?:...] punctuation
• [:blank:] [ t] space and TAB characters only
• [:space:] [ tnrfv]blank (whitespace) characters
• [:cntrl:] control characters
• [:graph:] [^ tnrfv] printed characters
• [:print:] [^tnrfv] printed characters and space
• Example: [[:upper:]ab] should only match the uppercase letters and
lowercase 'a' and 'b'.

Grep
• grep print lines matching a pattern
(General Regular Expression Parser)
grep [options] PATTERN [FILES]
option
-c print number of output context
-E Interpret PATTERN as an extended regular expression
-h Supress the prefixing of filenames
-i ignore case
-l surpress normal output
-v invert the sense of matching
% grep in words.txt
% grep –c in words.txt words2.txt
% grep –c –v in words.txt words2.txt

Grep usage samples
$ grep <string> <file or directory>
$ grep 'new FooDao' Bar.java
$ grep Account *.xml
$ grep –r 'Dao[Impl|Mock]' src
• Quote string if spaces or regex.
• Recursive flag is typical
• Don't quote filename with wildcards!

Common grep options
Case-insensitive search:
$ grep –i foo bar.txt
Only find word matches:
$ grep –rw foo src
Display line number:
$ grep –nr 'new Foo()' src

Filtering results
Inverted search:
$ grep –v foo bar.txt
Prints lines not containing foo.
Typical use:
$ grep –r User src | grep –v svn
Using find … | xargs, grep … is faster.

More grep options
Search for multiple terms:
$ grep -e foo –e bar appdata.txt
Find surrounding lines:
$ grep –r –C 2 foo logs
Similarly –A or –B will print lines before and after the line containing match.

G‟RE‟P
• Unix great utility is grep it uses regular expression
• Search for lines ending with “e”
% grep e$ words2.txt
• Search for “a”
% grep a[[:blank:]] word2.txt
• Search for words starting with “Th.”
% grep Th.[[:blank:]] words2.txt
• Search for lines with 10 lower case characters
% grep –E [a-z]{10} words2.txt
148
shell programming regex

Grep summary
• -r recursive search
• -i case insensitive
• -w whole word
• -n line number
• -e multiple searches
• -A After
• -B Before
• -C Centered

Unix
Find
NAME, Types, SIZE, MTIME,
CTIME, MMIN, PERM
Command

Basic find examples
$ find . –name Account.java
$ find /etc –name '*.conf'
$ find . –name '*.xml'
$ find . -not -name '*.java' -maxdepth 4
$ find . (-name '*jsp' –o –name '*xml')
• -iname case-insensitive
• ! == -not
• Quotes keep shell from expanding wildcards.

Find and do stuff
$ find . –name '*.java' | xargs wc –l | sort
Other options:
$ find . –name '*.java' -exec wc –l {} ; | sort
$ find . –name '*.java' -exec wc –l {} + | sort
Use your imagination. mv, rm, cp, chmod . . .

-exec vs | xargs
• -exec has crazy syntax.
• | xargs fits Unix philosophy.
• ; is slow, executes command once for each line.
• ; not sensible, sorts 'alphabetically.'
• | xargs may fail with filenames containing whitespace, quotes or slashes.

Find by type
Files:
$ find . –type f
Directories:
$ find . –type d
Links:
$ find . –type l

Find by modification time
Changed within day:
$ find . –mtime -1
Changed within minute:
$ find . –mmin -15
Variants –ctime, -cmin, -atime, -amin aren't especially useful.

Find Latest file
Compare to file
$ find . –newer foo.txt
$ find . ! –newer foo.txt

Find by permissions
$ find . –perm 644
$ find . –perm –u=w
$ find . –perm –ug=w
$ find . –perm –o=x

Find by size
Less than 1 kB:
$ find . –size -1k
More than 100MB:
$ find . –size +100M

Process
Management
Process Life, termination, scheduling
jobs

Process management
• Process monitoring
• Mechanism of Process Creation
• Internal and External commands
• Process States and Zombies
• Running Jobs in Background
• nice: Job execution with Low Priority
• Killing Processes with Signals
• Job Control
• at and batch: Execute Later
• cron: Running Jobs Periodically

View the Processes
• ps: A process is an executing program identified
by a unique PID (process identifier).
• To see information about your processes, with their
associated PID and status, type:
– $ ps
Options:
-e Select all processes.
-f Show columns user, pid, ppid, cpu, stime, tty, time, and args, in that
order.
-l Show columns flags, state, uid, pid, ppid, cpu, intpri, nice, addr, sz,
wchan, tty, time, and comm, in that order.
-x Shows the command line in extended format.

Process Life
• Kill command will pass the signals

Background jobs
• To background a process, type an & at the end of the
command line.
For example, the command sleep waits a given number of seconds
before continuing. Type
$ sleep 10
• This will wait 10 seconds before returning the command prompt $.
Until the command prompt is returned, you can do nothing except
wait.
To run sleep in the background, type:
$ sleep 10 &
• The & runs the job in the background and returns the prompt
straight away, allowing you do run other programs while waiting for
that one to finish.

• nohup: The Kernel terminates all active
processes by sending a hang-up signal at the
time of logout. The nohup command is used
to continue execution of a process even after
the user has logged out.
Syntax:
nohup <command> <Enter>
• Any command that is specified with
nohup, will continue running even after the
user has logged out.

• Chaining multiple commands:
$ command1;command2;command3
• You can suspend the process running in the foreground by
holding down the [control] key and typing [z] (written as ^Z)
Then to put it in the background, type
$ bg
Note: background programs that do not require user
interaction
• top:
Display and update information about the top processes on
the system.

• To see the background jobs running in the terminal:
$ jobs
• To restart (foreground) a suspended processes, type
$ fg %jobnumber
• For example, to restart sleep 100, type
$ fg %1
• Typing fg with no job number foregrounds the last
suspended process.

kill – Terminate Process
• Occasionally a need may arise to stop the process
adhoc
• This is done by sending the software, a termination
signal to kill the process.
$ kill <pid>
• Some programs are designed to ignore the interrupts.
– We can request the kill command to send a sure kill signal.
– The sure kill signal is represented by including a minus nine
option (-9) along with the kill command.
$ kill –9 1020
• Only the owner of the process or the super user can
kill a process.

Monitoring processes
• ps
• ps –ef
• ps –u oracle
• ps –C sshd
• man ps

Cron Execution example

Scheduling Jobs
% crontab –l # listing schedular
% crontab –e # editing cron
0 0 * * * daily-midnight-job.sh
0 * * * * hourly-job.sh
* * * * * every-minute.sh
0 1 * * 0 1AM-on-sunday.sh
% EDITOR=vi crontab –e
% man 5 crontab

Resource
Management
Disk space, RAM, Swap, top,
uptime

Resource Management
• Disk file usage : df
• Diskspace utilization : du
• Memory space with top, vmstat
• Swap space with free
• CPU uptime

Disk file usage
• df [options] [resource]
• df - summarize disk block and file usage
• df is used to report the number of disk blocks and i-nodes used and free for
each file system. The output format and valid options are very specific to the
OS and program version in use.
• -l local file systems
• -k kilobytes
• -h human readable form output

The df example
• df
Reports on the space left on the file system.
Command Format:
df [options] [special directory]
Example:
$ df –k ~  Will print information about the home directory.
• bdf
Developed by Berkeley. It displays the free size in a more presentable manner.

Disk space utilization
• du - report disk space in use
• du reports the amount of disk space in use for the files or directories you
specify.
• du [options] [directory or file]
• -a display disk usage for each file, not just subdirectories
• -s display a summary total only
• -k report in kilobytes (SVR4)
Outputs the number of kilobyes used by each subdirectory. Useful if you have
gone over quota and you want to find out which directory has the most files.
Example:
$ du -a

RAM size with free -m
• The free -m

The cpu load average
• The uptime command will give you load averages on CPU

Network
telnet, ftp, finger, ssh, scp, rsync

Network
• telnet and ftp
• Communication check : finger
• Remote files : scp, rcp, rsync
• Download from command wget
• Upload from command curl
• Email alerts
• Remote script : ssh
• Password less connectivity

telnet and ftp
• TELNET and FTP are Application Level Internet protocols. The TELNET
and FTP protocol specifications have been implemented by many different
sources, including The National Center for Supercomputer Applications
(NCSA), and many other public domain and shareware sources which are
also TELNET protocol implementations
– telnet [options] [ remote_host [ port] ]
– ftp [options] [ remote_host ]

ftp options
• ftp telnet Action
• -d set debugging mode on
• -d same as above (SVR4 only)
• -i turn off interactive prompting
• -n don't attempt auto-login on connection
• -v verbose mode on
• -l user connect with username, user, on the remote host

Example of ftp
ftp –n –u server.mail.edu <<_FTP_
user username password
put FILE
_FTP_

Download from command
• wget
ftp://user:pass@server.wfu.edu/file
• wget –r
ftp://user:pass@server.wfu.edu/dir/

Upload with curl, ftp
curl –T upload-file
-u username:password
ftp://server.wfu.edu/dir/file

Communication finger
• finger displays the .plan file of a specific user, or reports who
is logged into a specific machine. The user must allow general
read permission on the .plan file.
• finger [options] [user[@hostname]]
• Common Options
• -l force long output format
• -m match username only, not first or last names
• -s force short output format

Remote script SSH
• The Secure SHell (SSH) versions of the rcp, rsh, and
rlogin programs are freely available and provide much
greater security
Command Format:
$ rlogin <host name> -l <user name>
$ rlogin baileys -l snadmin
• Password less ssh connection to remote server
– ssh –t rsa

Email
• Email Notification
% echo “Message” |
mail –s “Here’s your message”
bhavanishekhar@gmail.com

Smart directory copy
with rsyncrsync (remote sync) has been designed to keep in sync
directories on 2 machines with a low bandwidth connection.
Only copies files that have changed. Files with the same size are
compared by checksums.
Only transfers the blocks that differ within a file!
Can compress the transferred blocks
Preserves symbolic links and file permissions: also very useful for copies
on the same machine.
Can work through ssh (secure remote shell). Very useful to update the
contents of a website, for example.

rsync examples (1)
rsync -a /home/arvin/sd6_agents/ /home/sydney/misc/
-a: archive mode. Equivalent to -rlptgoD... easy way to tell you want
recursion and want to preserve almost everything.
rsync -Pav --delete /home/steve/ideas/ /home/bill/my_ideas/
-P: --partial (keep partially transferred files) and --progress (show
progress during transfer)
--delete: delete files in the target which don't exist in the source.
Caution: directory names should end with / . Otherwise, you get a
my_ideas/ideas/ directory at the destination.

rsync examples (2)
Copying to a remote machine
rsync -Pav /home/bill/legal/arguments/
bill@www.sco.com:/home/legal/arguments/
User bill will be prompted for a password.
Copying from a remote machine through ssh
rsync -Pav -e ssh homer@tank.duff.com:/prod/beer/
fridge/homer/beer/
User homer will be prompted for his ssh key password.

Q & A
• Summary
• References
– Middlewareadmin.net
– Wlatricksntips.blogspot.com
• Feature Automations skills
– AWK
– SED
– PERL
– Python

Unix shell scripting

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Unix shell scripting

Ähnlich wie Unix shell scripting (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Unix shell scripting

Hinweis der Redaktion