Ever had a problem? No one could help you? Please call the BASH Team! This presentation starts with some preliminary thoughts about software integration. Shell scripting means knowing your system and the applications you have available that can do the dirty stuff for you. BASH is all about gluing... so what are the ways we can use to control resources and integrate software through scripting? Some set of "advanced" features of BASH are explained and I'm sure you'll find them very handy to enhance your scripts. The BASH recipes can be used for common daily tasks of end users or administrators. "Hacking" web sites can also be done using BASH and some command line tools. We'll talk a bit about it. Learn also with real examples explained, namely: - fetch music streams from web URL's to disk, from a popular Portuguese radio station - emulate a user/replicate online banking tasks and extract values automatically, with a real bank example (by means of "reverse engineering" web pages) - send SMS from command line, using a mobile operator's site (by means of "reverse engineering" web pages) This is not intended to be a BASH tutorial for beginners, neither a simple plain tutorial. It's more a kind of BASH "hacking" guide.

1. If you have a problem, if no one else can help... and if you can find them, maybe you can hire the bAsh-TEAM A different BASH shell scripting tutorial 03.12.2009 @ Codebits 2009 Thisdocumentisintellectualpropertyof PT Inovação, SA andshouldnotbeused without express written permission.

2. Schedule/Agenda A little about me Preliminary thoughts Pragmatic programming & Gluing stuff BASH Some characteristics Tricky snippets explained Introduction to “Hacking” web sites (tools, how-to) Some examples explained Extra stuff (if we have time, I’ll guess we’ll have) References/Further reading 03-12-2009 2

3. Warm-up Random crap… 03-12-2009 3

4. A little about me Co-founder of Radioactive Design portuguese demogroup in the 90’s Experience in “hacking” mostly in Linux, networking and protocols (messaging protocols@Layer 7, TCP@Layer 4) I love problems and to find their sources! Doing work for a “telco” R&D (PT Inovação), developing and managing projects and sometimes doing research  Languages Java (mostly J2SE) Perl Ruby / Ruby-on-Rails BASH Some C and if you dig a little you’ll find also Pascal and even Visual Basic  03-12-2009 4

5. Preliminary thoughts Pragmatic programming Don’t try to reinvent the wheel, at least every week  Be pragmatic: If you can reuse a solution, think about it! If you’re going to implement it, and if it really doesn’t matter what technology you should be using… then use the fastest/pragmatic approach; don’t implement it in your preferred language just because you like more that one 03-12-2009 5

6. Gluing stuff Make it all work together Some call it gluing, other prefer “integration”… I just say make it all work Integration can be made using well defined API’s or some “out-of-band glue” External Scripting analyzing logs, produced files, etc… Know your system! Be aware of the tools available in your OS, they will make your life a lot easier! 03-12-2009 6

7. Typical Scripting Flow 03-12-2009 7 catfile.txt | egrep -v “[0-9]”| sort | uniq | xargstouch ….

8. BASH What about… 03-12-2009 8

9. BASH: Bourne-Again Shell BASH it’s not just a simple shell, it’s not just a scripting language nor a simple application BASH is all about integrating software and controlling resources through scripting 03-12-2009 9 Once again, know your distribution!

10. Gluing with BASH Passing data/information all around Data can be exchanged within hosts or between hosts, using sockets or any other messaging passing method In BASH programs communicate using: Standard streams (stdin, stdout, stderr) The programs exit/return code Signals And sometimes using environment variables The communication can be made by any kind of “out-of-band”protocol” (socket, some file, …) 03-12-2009 10

11. Successvs Failure Thereversedtruth: thebooleanupsidedown A successfulprogramterminationreturnstheexitcode 0; therefore “true” in BASH is 0 Allabnormalprogramterminationreturns a valuediferentof 0 to the shell; therefore “false” in BASH iseverythingexcept 0 Note thatinalmost 100% oflanguages, as inlogic, “false” isrepresentedby 0 and “true” byanyothervalue 03-12-2009 11

12. (Some) BASH specialvariables $$ Current PID of the running program $! PID of last child process started by the current shell $? Exit code of the last executed program !$ Arguments used in the last command $0 Name of current running script $1..$n First to n-th argument passed to the script 03-12-2009 12

13. RedirectingStreams Streamscanberedirected Youcanmakestdoutpoint to some file descriptororyoucanmakestderrpoint to thesame FD usedbystdout Cautionwiththeorderofredirection: theredirectionismadenot as a wholebutfromleft to right > some_file Redirectsstdoutto some_file 2>&1 Redirectsstderr to thesame FS usedby 2>&- Redirectsstderr to null (/dev/null) > bla 2>&1isdifferentfrom2>&1 > bla 03-12-2009 13

14. Handling signals A BASH script, as any standard program, can catch and handle signals! You can do this to provide things such as configuration reloading or graceful exits 03-12-2009 14 $LOCK=/tmp/somefile.lock [ -f $LOCK ] && exit 0 trap "{ rm -f $LOCK ; exit 255; }" EXIT touch $LOCK || exit 255 ….

15. Paralell Execution Multiple programs can be started in the background, by appending “&” in the command line “wait” can be used to wait for child processes and therefore provide basic means of process synchronization; a PID argument can be given 03-12-2009 15 find /home/ -type d | do_some_weird_stuff& another_command& wait echo “yes, finally I can shutdown!! ” ; shutdown –hnow

16. Subshells Another shell can be instantiated within the current shell (the subshell has its own PID) A subshell is delimited by parentheses You can use a subshell to execute several programs/commands and “grab” the whole output (stdout/stderr) produced in the context of that subshell to further process it! 03-12-2009 16 (find /home/ -type d;ls /home/sergio) | grepgold

17. Tricky snippeTs Going deep through and hacking with… 03-12-2009 17

18. Introduction to “Hacking” Web sitesTools A protocol analyzer is your friend (e.g. Wireshark / tshark) but it won’t deal with HTTPs You can use Firefox plus some useful extensions no analyze HTTP(s) requests: Live HTTP headers Tamperdata Modify Headers In most Linux distributions you have HTTP clients that can be easily integrated with BASH GET/POST wget (preferred) 03-12-2009 18

19. Introduction to “Hacking” Web sitesHow-to (1/2) View the source of web pages, they’ll reveal much of the web site logic (and you’ll find many surprising things!) Sites often do validation in the client side using JavaScript => security problems,hidden or extended features 03-12-2009 19

20. Introduction to “Hacking” Web sitesHow-to (2/2) Many parameters inside HTML are pure garbage Some sites demand HTTP requests to be made using POST while for other it’s indifferent Some sites do redirect on POST! So your client must either implement this behavior or else POST to the final URL 03-12-2009 20

21. Fetchandsavemusicstreams 03-12-2009 21 pnumber[1]=2904; pname[1]="Antena3DanceClub”; pnumber[2]=1078; name[2]="AmbientaSons“; max=2; basedir="antena3“; totchilds=0; lastnchilds=-1 for n in`seq 1 $max`; do echo "processing${pname[$n]}...”; mkdir -p "$basedir/${pname[$n]}“ wmafiles=`GET -P "http://ww1.rtp.pt/multimedia/programa.php?hist=1&prog=${pnumber[$n]}" | egrep -o "mms://.*wma"|sort|uniq` for wma in $wmafiles; do file=`basename "$wma"` filepath="$basedir/${pname[$n]}/$file“ fprocs=`lsof$filepath2>- | wc -l 2>-` if [ "$fprocs" -gt 0 ] then echo "someone is accessing $file... bypassing.." else echo "fetching$file..." nohupmimms -r "$wma" "$filepath" >- 2>&1 & ((totchilds++)) fi done done echo "waiting for background processes to finish fetching all musics...“ wait

22. Send SMS using a mobile operator’s site 03-12-2009 22 if [ $# -le 0 ] then echo "wrong syntax. use vod.sh dest message" exit fi user=9100000; pass=“yourpassword”; dest=$1 ; msg=$2 rm -f cookies.txt wget -O - --keep-session-cookies --save-cookies cookies.txt "http://www.vodafone.pt/main/myVodafone/" >- wget -O - --load-cookies cookies.txt --keep-session-cookies --save-cookies cookies.txt "--post-data=userid=$user&password=$pass&sru=https%3A%2F%2Fmy.vodafone.pt%2Fpm%2FSPMDispatcher.aspx%3FPMcmd%3D17%26userClass%3D10%26Guid%3D%7BB9DED956-B87F-4C24-852C-70A3BBBB0161%7D%26ReturnUrl%3Dhttps%253a%252f%252fmy.vodafone.pt%252fguest%252fhomepagePre.htm&fru=https%3A%2F%2Fmy.vodafone.pt%2Fguest%2FhomepagePre.htm&svc_id=myprodpub" “https://id.vodafone.pt/ucp//auth/login.asp?&ou=&crypt=0&prf=0&key=alias“ >- wget -O - --load-cookies cookies.txt --keep-session-cookies --save-cookies cookies.txt "http://mysms.vodafone.pt/rules/sms/sms_envio.asp" >- wget -O - --load-cookies cookies.txt --keep-session-cookies --save-cookies cookies.txt "--post-data=mydate3=&indicativo=91&telefone=&mensagem1=$msg&programado=nao&h2=1&phones=$dest%2F&prog=&dataf=12%2F29%2F2009+10%3A58%3A23+PM&agt=" br /> "--referer=http://mysms.vodafone.pt/rules/sms/sms_envio.asp" “http://mysms.vodafone.pt/rules/sms/sms_envio.asp?submit=ok” >-

23. Accessingan online Bankandgetting data 03-12-2009 23 datainicial="2009-11-01" maxdias=$((365*5)) hoje=`date +%Y-%m-%d` fundo="BPI Portugal" for i in`seq 0 $maxdias`; do data=`date +%Y-%m-%d -d "$datainicial + $i day"` if [ "$data" == "$hoje" ]; thenbreakfi readvaldt< <(GET "http://www.bpiinvestimentos.pt/Fundos/QuadroCotacoesfundos.asp?opc=1&DataPesquisa=$data" | grep -A 2 "$fundo"| egrep -o "[0-9]{2}-[0-9]{2}-[0-9]{4}|[0-9]{1,2},[0-9]{1,}" |tr "" " "| awk '{print $2,$1}‘ ) val=$(echo$val | tr "," ".") dt=$(echo$dt | tr "-" " "| awk '{print $3,$2,$1}' | tr " " "-") psql -q -h localhost -U postgres -c "deletefromcotacoeswherefundo='$fundo' anddata='$dt'" fundos psql -q -h localhost -U postgres -c "insertintocotacoes (fundo,data,valor) values ('$fundo','$dt',$val)" fundos done

24. EXTRA STUFF If we have time, we gonna take a bit of… 03-12-2009 24

25. “here” / inlinedocuments A heredocumentallowsthespecificationof a blockofthatcanbefed to a variable, to BASH functionsordirectly to the STDIN of a program 03-12-2009 25 multiline_variable=$(cat<<END_DELIMITER a b END_DELIMITER) echo “$multiline_variable” ./some_app<<SEQUENCE_OF_APP_COMMANDS do this do that … SEQUENCE_OF_COMMANDS

26. Processsubstitution “A kind of a reverse pipe” the output of a command (list) is redirected to a temporary file descriptor which can be used as the STDIN (directly or indirectly through an argument) of a program 03-12-2009 26 echo<( ls / ) /dev/fd/63 cat<( ls / ) /home … # thefollowinglinewouldbe “hard” to do withpipes, unlessusing a subshelllike (ls /; ls /home) | grep “o” grep “o” <( ls / ) <( ls /home) /dev/fd/63:boot /dev/fd/63:cdrom /dev/fd/62:sergio /dev/fd/62:sergio …

27. References/Further reading AdvancedBash-ScriptingGuide http://tldp.org/LDP/abs/html/ Linux Shell Scripting Tutorial - A Beginner's handbook http://bash.cyberciti.biz/guide/Main_Page Snipt http://snipt.net/public/tag/bash Bash by example, IBM developerWorks http://www.ibm.com/developerworks/library/l-bash.html 03-12-2009 27

28. Thanks / Obrigado By: Sergio Freire Mensagens e RedesConvergentes / Departamento de Redes e Protocolos Tel: +351 234 403609 sergio-s-freire@ptinovacao.pt Thisdocumentisintellectualpropertyof PT Inovação, SA andshouldnotbeused without express written permission.