Présentation aux Geeks Anonymes Liège par Cyril Soldani, le 13 décembre 2017. Page des Geeks Anonymes : https://www.recherche.uliege.be/cms/c_9463913/fr/geeks-anonymes

1. SHELL SCRIPTING
Cyril Soldani
cyril.soldani@uliege.be

2. OUTLINE
What is it all about?
Why should I care?
A 5-minute introduction to POSIX shell scripts
How not to shoot yourself in the foot
A few recipes?
Discussion

3. WHAT IS IT ALL ABOUT?

4. Interactive: Script:
WHAT IS A SHELL SCRIPT?
A shell is a command-line UI to access OS services.
A shell script is a sequence of shell commands.
$ mkdir a_folder
$ cd a_folder
$ echo "Hello"
Hello
$ for n in John Mary; do
for> echo $n
for> done
John
Mary
#!/bin/sh
mkdir a_folder
cd a_folder
echo "Hello"
for n in John Mary; do
echo $n
done

5. THERE ARE AS MANY SHELL
SCRIPTING LANGUAGES AS
SHELLS
UNIX shells: bash, zsh, ksh, tcsh, ...
Microso shells: command.com, PowerShell.
Most UNIX shells support a standard: POSIX.
bash and POSIX are the most used.
Windows 10 now supports bash (and POSIX).

6. WHY SHOULD I CARE?

7. THE SHELL STILL MATTERS
It is more suited than the GUI for many tasks, e.g.
deleting a set of files following a given pattern:
It allows automation through scripting.
Let the machine do the work for you!
It is still used in many places:
initialization scripts, application wrappers, tests,
administration tools, cron jobs, etc.
It allows easy remote administration.
rm *foo*.log *bar*.aux

8. SHELL SCRIPTS CAN BE SHORT
Get login dates and users according to systemd.
Shell scripts are generally the quickest way to
automate a simple task.
grep 'systemd.*Slice' /var/log/syslog | cut -d' ' -f1-3,11
import re
pat = re.compile('systemd.*Slice')
with open("/var/log/syslog") as f:
for line in f:
if pat.search(line):
words = line.rstrip().split(' ')
print(words[0], words[1], words[2], words[10])

9. IT IS BEGINNER-FRIENDLY
Learning the shell is useful.
Basic shell use is easy to learn.
Shell scripting comes free if you know the shell!
... but it is hard to master :'(

10. A 5-MINUTE INTRODUCTION TO
POSIX SHELL SCRIPTS
We all know that time is relative.

11. A SIMPLE SCRIPT
#!/bin/sh
rate=4000 # Estimated disk rate in kB/s
while sleep 1; do
dirty=$(grep 'Dirty:' /proc/meminfo | tr -s ' '
| cut -d' ' -f2)
if [ $dirty -lt 1000 ]; then
break;
fi
secs=$((dirty / rate))
hours=$((secs / 3600)); secs=$((secs - 3600 * hours))
mins=$((secs / 60)); secs=$((secs - 60 * mins))
printf "r%02dh %02dm %02ds" $hours $mins $secs
done
echo

12. REDIRECTIONS
You can use other file descriptors than 1 (stdout)
and 2 (stderr) to make crazy stuff.
The order can be tricky. The engineer way: test it.
cmd1 | cmd2 # cmd2.stdin = cmd1.stdout
cmd > out.txt # File out.txt = cmd.stdout
cmd 1> out.txt # Idem
cmd >> out.txt # cmd.stdout is appended to out.txt
cmd < in.txt # cmd.stdin = contents of file in.txt
cmd < in.txt > out.txt # You can combine
cmd 2> /dev/null # cmd.stderr written to /dev/null
cmd > out.txt 2>> err.txt
1>&2 cmd # cmd.stdout will go to stderr

13. ANOTHER SIMPLE SCRIPT
#!/bin/sh
die() {
>&2 echo "$1"
exit 1
}
[ $# -eq 1 ] || die "Usage: lumount DISK-LABEL"
[ -L "/dev/disk/by-label/$1" ] ||
die "No disk with label '$1'."
[ -d "/media/$1" ] ||
die "Cannot find '/media/$1' mount point."
pumount "$1"

14. HERE DOCUMENTS
cmd <<ZORGLUB
All file content up to next ZORGLUB on its own line is fed
to cmd.stdin.
Note that you can use ${variable}s and
embedded $(echo "commands").
ZORGLUB
#!/bin/sh
# Usage: dhcp_add_machine MAC_ADDRESS HOSTNAME
cat <<EOF >> /etc/dhcp/dhcpd.conf
host $2 {
fixed-address $2.run.montefiore.ulg.ac.be;
hardware ethernet $1;
}
EOF
systemctl restart dhcpd.service

15. PARAMETER EXPANSION
You can modify variable content in various ways
directly from the shell.
${foo:-bar} # $foo if defined and not null, "bar" otherwise
${foo:+bar} # null if $foo is unset or null, "bar" otherwise
${foo%bar} # removes smallest "bar" suffix from foo
${foo%%bar} # removes largest "bar" suffix from foo
${haystack/pin/needle} # substring replacement (not POSIX)
rate=${1:-4000} # Initialize rate to first argument or 4000
if [ -z ${TARGET_DIR:+set} ]; then
die "TARGET_DIR is not specified!"
fi
${filename%%.*} # removes all extensions from filename
for i in *CurrentEdit*; do
mv "$i" "${i/CurrentEdit/_edit_}"
# trucCurrentEdit42.log -> truc_edit_42.log
done

16. HOW NOT TO SHOOT YOURSELF
IN THE FOOT
... or anywhere else where it hurts.

17. WHAT IS WRONG WITH THAT
SCRIPT?
The correct path is /var/srv/mightyapp/tmp
(without dash).
cd will fail, but the script will continue and erase
everything in current directory!
#!/bin/sh
# Clean-up mighty-app temporary files
if pgrep mighty-app >/dev/null; then
>&2 echo "Error: mighty-app is running, stop it first!"
exit 1
fi
cd /var/srv/mighty-app/tmp
rm -rf *

18. ALWAYS USE set -e
The script will fail if any command fail.
You can still use failing commands in conditions,
and with ||.
You can temporarily disable checking with set +e.
#!/bin/sh
set -e
...

19. THE PIPE ABSORBS ERRORS!
faulty_cmd fails, grep might fail, but cut will be
happy, and critical_processing will be called
with bogus data!
bash has an option set -o pipefail for this, but
beware of broken pipes:
#!/bin/sh
set -e
data=$(faulty_cmd | grep some_pattern | cut -d' ' -f1)
critical_processing -data $data
#!/bin/bash
set -eo pipefail
first=$(grep "systemd.*Slice" /var/log/syslog | head -n1)
# The above will fail if head exits before grep

20. WHAT IS WRONG WITH THAT
SCRIPT?
$MIGHTY_APP_TMP_DIR might be undefined.
cd will happily go to your home folder...
... where every file will be deleted!
#!/bin/sh
set -e
# Clean-up mighty-app temporary files
if pgrep mighty-app >/dev/null; then
>&2 echo "Error: mighty-app is running, stop it first!"
exit 1
fi
cd "$MIGHTY_APP_TMP_DIR"
rm -rf *

21. ALWAYS USE set -u
The script will fail if it tries to use an undefined
variable.
You can test for definition using parameter
expansion.
You can use set +u to disable temporarily, e.g.
before sourcing a more laxist file.
#!/bin/sh
set -eu
...

22. A FALSE GOOD IDEA
It is shorter and cleaner, right?
No! What if someone does sh your_script.sh?
#!/bin/sh -eu
...

23. QUOTE LIBERALLY!
cd $1 is expanded to cd My project.
Use cd "$1" instead.
#!/bin/sh
# Add given directory to source control
set -eu
cd $1
git init
...
$ git-add-dir "My project"
git-add-dir: 3: cd: can't cd to My
zsh: exit 2 git-add-dir "My project"

24. USING COMMANDS WHICH ARE
NOT AVAILABLE
As most shell scripting commands are actually
programs, they must be installed (and in path).
Beware of what you assume will be there.
Commands might also behave differently than what
you expect (e.g. ps on Linux behaves much
differently than the one on FreeBSD).

25. RELYING ON VARIABLE COMMAND
OUTPUT
If you process the ouptut of some commands in your
scripts, ensure that the command output is well-
defined, and stable.
E.g. the output of ls will vary from system to sytem,
between versions, and even depends on shell and
terminal configuration! Use find instead.
Some commands have flags to switch from a
human-readable format to one that is easily
processed by a machine.

26. BEWARE OF SUBPROCESSES
The right-hand side of | runs in a subprocess:
Use FIFOs or process susbstitution (bash) instead:
maxVal=-1
get_some_numbers | while read i; do
if [ $i -gt $maxVal ]; then
maxVal=$i
fi
done
# $maxVal is back to -1 here
maxVal=-1
while read i; do
if [ $i -gt $maxVal ]; then
maxVal=$i
fi
done < <(get_some_numbers)

27. MODIFYING THE OUTER
ENVIRONMENT
A script runs in its own process, it cannot modify the
caller environment (export variables, define
functions or aliases).
Source the file with . instead, it will be included in
the running shell.
$ pyvenv my-venv # Creates a python virtual environment
$ sh my-venv/bin/activate # Does nothing
$ . my-venv/bin/activate
(my-venv) $

28. HANDLING SIGNALS
Clean-up a er yourself ...
... even if something strange occurred!
tmpdir=$(mktemp -d "myscript.XXXXXX")
trap 'rm -rf "$tmpdir"' EXIT INT TERM HUP
...

29. IN SUMMARY
Always use set -eu.
Quote liberally.
Program defensively.
Think about your dependencies.
Think about what runs in which process.
Clean-up a er yourself.
Test, test, test.

30. A FEW RECIPES

31. WRAPPER SCRIPTS
That pesky program keeps trying to open ou i files
with trucmuche instead of tartempion? Put the
following in bin/trucmuche:
You want that huge terminal font size for beamers?
is all you need.
#!/bin/sh
set -eu
exec tartempion "$@"
#!/bin/sh
set -eu
exec urxvt -fn xft:Mono:size=24 "$@"

32. DEPLOYMENT HELPERS
You want to package that Java application so that it
looks like a regular program?
#!/bin/sh
set -eu
exec java -jar /usr/libexec/MyApp/bloated.jar
my.insanely.long.package.name.MyApp "$@"

33. FOLLOWING A FILE
How to process a log file with a shell script so that it
continues its execution each time a new line is
appended to the log?
#!/bin/bash
set -euo pipefail
process_line() {
# Code to process a line...
}
while read line; do
process_line "$line"
done < <(tail -n +1 -f /var/log/my_log)

34. USING ANOTHER LANGUAGE
Use here scripts for short snippets:
Or delegate to another interpreter entirely:
Changing the shebang might be all that's required!
#!/bin/sh
myVar=$(some_command -some-argument)
python3 <<EOF
print("myVar =", $myVar) # Note the variable substitution!
EOF
#!/bin/sh
"exec" "python3" "$0" "$@" # Python ignores strings
print("Hello") # Python code from now on
#!/usr/bin/env python3
print("Hello")

35. DISCUSSION

36. TO BASH, OR NOT TO BASH?
bash pros:
More expressive (e.g. arrays, process
substitution).
Safer (e.g. pipefail).
POSIX pros:
More portable.
Faster execution.
Mostly forces you to stick to simple tasks!

37. SHELLSCRIPTOR VS PYTHONISTA
#!/bin/sh
set -eu
LLVM_CONFIG=${LLVM_CONFIG:-llvm-config-4.0}
CFLAGS="$($LLVM_CONFIG --cflags) ${CFLAGS:-}"
LDFLAGS="$($LLVM_CONFIG --ldflags) ${LDFLAGS:-}"
LDLIBS="$($LLVM_CONFIG --libs) ${LDLIBS:-}"
CC="${CC:-clang}"
for src in *.c; do
$CC $CFLAGS -c -o "${src%.c}.o" "$src"
done
$CC -o compiler *.o $LDFLAGS $LDLIBS

38. SHELLSCRIPTOR VS PYTHONISTA
#!/usr/bin/env python3
import glob, os, subprocess
def config(arg):
llvm_config = os.environ.get("LLVM_CONFIG", "llvm-config-4.0")
return subprocess.check_output([llvm_config, arg]).decode('utf-8')
cflags = config("--cflags") + " " + os.environ.get("CFLAGS", "")
ldflags = config("--ldflags") + " " + os.environ.get("LDFLAGS", "")
libs = config("--libs") + " " + os.environ.get("LDLIBS", "")
cc = os.environ.get("CC", "clang")
for src in glob.glob("*.c"):
obj = src.rsplit(".", 1)[0] + ".o"
subprocess.check_call([cc, cflags, "-c", "-o", obj, src])
objs = glob.glob("*.o")
subprocess.check_call([cc, "-o", "compiler"] + objs + [ldflags, libs])

39. TO SHELL-SCRIPT, OR NOT TO
SHELL-SCRIPT?
Pros:
Quick way to automate simple tasks.
You (should) already know it.
Always available.
Good for one-shots.
Cons:
Hard to write reliable scripts.
Limited expressivity.
Shell scripts can quickly become cryptic.
Manual dependency tracking.
Limited reuse.