Sînt descrise programele libele GNU Parallel și GNU Stow (opțiuni, exemple de utilizare, contraexemple etc)

1. GNU Parallel s, i GNU Stow
Ziua Libert˘at, ii Programelor s, i 30 de ani de GNU ˆın B˘alt, i
Radu Dumbr˘aveanu
Universitatea de Stat “A. Russo” din B˘alt, i, Republica Moldova
B˘alt, i, 2013

2. GNU Parallel
GNU parallel este un program cu interfat, ˘a ˆın linia de comand˘a
(CLI) care permite utilizatorului executarea ˆın paralel a unei
comenzi sau a unui set de comenzi (script shell).

3. GNU Parallel
GNU parallel este un program cu interfat, ˘a ˆın linia de comand˘a
(CLI) care permite utilizatorului executarea ˆın paralel a unei
comenzi sau a unui set de comenzi (script shell).
Sintaxa:
parallel [optiuni] [comanda [argumente]] < lista de argumente
sau
parallel [optiuni] [comanda [argumente]] ::: argumente

4. Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel

5. Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org

6. Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org

7. Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org
find . −type f | parallel gzip

8. Exemple clasice de utilizare
(echo ls; echo grep root /etc/passwd) | parallel
parallel traceroute ::: foss.org.my debian.org freenetproject.org
parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org
find . −type f | parallel gzip
ls ∗.gz | parallel −X mv {} archive

9. Dificult˘at, i?

10. Dificult˘at, i?
Lansarea oric˘arei comenzi ce cont, ine parallel se sfˆırs, es, te cu
mesajul de eroare:
The program ’parallel’ is currently not installed. You can
install it by typing:
sudo apt−get install moreutils

11. Dificult˘at, i?
Lansarea oric˘arei comenzi ce cont, ine parallel se sfˆırs, es, te cu
mesajul de eroare:
The program ’parallel’ is currently not installed. You can
install it by typing:
sudo apt−get install moreutils
Nu urmat, i acest˘a indicat, ie; comanda parallel din pachetul
moreutils este cu totul altceva.

12. Modalit˘at, i de instalare
Putem instala parallel ˆın felul urm˘ator:
1 sudo apt−get install parallel

13. Modalit˘at, i de instalare
Putem instala parallel ˆın felul urm˘ator:
1 sudo apt−get install parallel
sau
1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz2
2 tar xf parallel−20130922.tar.bz2
3 cd parallel−20130922
4 ./configure
5 make
6 make install

14. Modalit˘at, i de instalare
Putem instala parallel ˆın felul urm˘ator:
1 sudo apt−get install parallel
sau
1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz2
2 tar xf parallel−20130922.tar.bz2
3 cd parallel−20130922
4 ./configure
5 make
6 make install
20130922 = versiunea parallel din 22.09.2013

15. Modalit˘at, i de instalare
Dac˘a nu sˆıntet, i root atunci:
creat, i dosarele ~/bin s, i ~/share apoi
1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz2
2 tar xf parallel−20130922.tar.bz2
3 cd parallel−20130922
4 ./configure −−prefix=$HOME
5 make
6 make install

16. Modalit˘at, i de instalare
Sau dac˘a dorit, i s˘a utilizat, i parallel f˘ar˘a a-l instala (de exemplu
pentru a experimenta):
1 wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
2 chmod 755 parallel
3 cp parallel sem
4 mv parallel sem ˜/bin/

17. Modalit˘at, i de instalare
Sau dac˘a dorit, i s˘a utilizat, i parallel f˘ar˘a a-l instala (de exemplu
pentru a experimenta):
1 wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
2 chmod 755 parallel
3 cp parallel sem
4 mv parallel sem ˜/bin/
parallel este un script Perl, nu ai ce compila.
Mai mult˘a informat, ie pe
https://github.com/razrichter/gnu_parallel.

18. Dificult˘at, i?

19. Dificult˘at, i?
Lansarea oric˘arei comenzi ce cont, ine parallel se sfˆırs, es, te cu
mesajul de eroare:
1 /bin/bash: {}: command not found
sau
1 /bin/bash: :::: command not found

20. Dificult˘at, i?
Lansarea oric˘arei comenzi ce cont, ine parallel se sfˆırs, es, te cu
mesajul de eroare:
1 /bin/bash: {}: command not found
sau
1 /bin/bash: :::: command not found
Solut, ii:

21. Dificult˘at, i?
Lansarea oric˘arei comenzi ce cont, ine parallel se sfˆırs, es, te cu
mesajul de eroare:
1 /bin/bash: {}: command not found
sau
1 /bin/bash: :::: command not found
Solut, ii:
ad˘augat, i opt, iunea --gnu (parallel --gnu ...);
sau sudo rm /etc/parallael/config;
sau http://stackoverflow.com/questions/16448887/
gnu-parallel-not-working-at-all

22. Exemple de utilizare (divizarea fis, ierelor mari pe blocuri)
cat fisiermare.sql | parallel −k −−pipe grep −i ”text”
cat bigfiledump.sql | parallel −k −−pipe −−block 2M
grep −i ”text”

23. Exemple de utilizare (divizarea fis, ierelor mari pe blocuri)
cat fisiermare.sql | parallel −k −−pipe grep −i ”text”
cat bigfiledump.sql | parallel −k −−pipe −−block 2M
grep −i ”text”
cat bigfiledump.sql | parallel −k −−pipe −−block 2M
−−sshlogin server.example.com,server2.example.net
grep −i ”text”

24. Modalit˘at, i de transmitere a parametrilor
Cum transmitem parametrii la comanda care trebuie executat˘a ˆın
paralel?

25. Modalit˘at, i de transmitere a parametrilor
Cum transmitem parametrii la comanda care trebuie executat˘a ˆın
paralel?
1) Prin intrarea standard:
1 seq 5 | parallel echo
2 1
3 2
4 3
5 4
6 5

26. Modalit˘at, i de transmitere a parametrilor
Accesul la parametri:
1 seq 5 | parallel echo ”parm={}”
2 param=1
3 param=2
4 param=3
5 param=4
6 param=5

27. Modalit˘at, i de transmitere a parametrilor
Mai mult, i parametri deodat˘a:
1 seq 10 | parallel −X echo
2 1 2 3 4 5
3 6 7 8 9 10

28. Modalit˘at, i de transmitere a parametrilor
2) Prin fis, iere:
1 parallel wget < adresse.lst
1 cat adrese.lst
2 foss.org.my
3 debian.org
4 freenetproject.org

29. Modalit˘at, i de transmitere a parametrilor
3) Prin linia de comand˘a:
1 parallel echo ::: 1 2 3 4
1 1
2 2
3 3
4 4

30. Modalit˘at, i de transmitere a parametrilor
3) Prin linia de comand˘a:
1 parallel echo ::: 1 2 3 4
1 1
2 2
3 3
4 4
sau
1 parallel unzip ::: ./∗.zip

31. Specificarea num˘arul de procese paralele
1) Num˘arul de procese = num˘arul de procesoare:
1 seq 10 20 | parallel −j+0 sleep
1 ps f −o pid,command
2 PID COMMAND
3 31379 −bash
4 32270 ps f −o pid,command
5 30463 −bash
6 30579 mc
7 30581 bash −rcfile .bashrc
8 32239 /usr/bin/perl −w /usr/bin/parallel −j+0 sleep
9 32268 sleep 10
10 32269 sleep 11

32. Specificarea num˘arul de procese paralele
2) Num˘arul maximal de procese, explicit:
1 seq 10 20 | parallel −j 5 sleep
1 ps f −o pid,command
2 PID COMMAND
3 31379 −bash
4 32523 ps f −o pid,command
5 30463 −bash
6 30579 mc
7 30581 bash −rcfile .bashrc
8 32487 /usr/bin/perl −w /usr/bin/parallel −j 5 sleep
9 32518 sleep 10
10 32519 sleep 11
11 32520 sleep 12
12 32521 sleep 13
13 32522 sleep 14

33. Specificarea num˘arul de procese paralele
3) Num˘arul de sarcini per procesor, explicit:
1 $ seq 10 20 | parallel −j 150% sleep
1 ps f −o pid,command
2 PID COMMAND
3 31379 −bash
4 32636 ps f −o pid,command
5 30463 −bash
6 30579 mc
7 30581 bash −rcfile .bashrc
8 32602 /usr/bin/perl −w /usr/bin/parallel −j 150% sleep
9 32632 sleep 10
10 32633 sleep 11
11 32634 sleep 12

34. Contraexemplu
1 cd dosarcu66925fisiere
2 /usr/bin/time cp −r ∗ /tmp
3 1.77user 49.62system 6:49.01elapsed 12%CPU
4 7692784inputs+7367600outputs (0major+320minor)

35. Contraexemplu
1 cd dosarcu66925fisiere
2 /usr/bin/time cp −r ∗ /tmp
3 1.77user 49.62system 6:49.01elapsed 12%CPU
4 7692784inputs+7367600outputs (0major+320minor)
1 cd dosarcu66925fisiere
2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp
3 2.40user 37.61system 6:30.38elapsed 10%CPU
4 7402904inputs+7367600outputs (21major+27017minor)

36. Contraexemplu
1 cd dosarcu66925fisiere
2 /usr/bin/time cp −r ∗ /tmp
3 1.77user 49.62system 6:49.01elapsed 12%CPU
4 7692784inputs+7367600outputs (0major+320minor)
1 cd dosarcu66925fisiere
2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp
3 2.40user 37.61system 6:30.38elapsed 10%CPU
4 7402904inputs+7367600outputs (21major+27017minor)
1 cd dosarcu66925fisiere
2 ls . | /usr/bin/time parallel −j 150% cp −r {} /tmp
3 2.51user 50.74system 12:55.95elapsed 6%CPU
4 7579504inputs+7367608outputs (27major+28524minor)

37. Adrese web
Pagina web
http://www.gnu.org/software/parallel
Manual s, i exemple
http://www.gnu.org/software/parallel/man.html
Sursele
http://savannah.gnu.org/git/?group=parallel

38. Adrese web
Tutoriale
http://savannah.gnu.org/projects/parallel
Video (partea I s, i II)
http://www.youtube.com/watch?v=OpaiGYxkSuQ
http://www.youtube.com/watch?v=P40akGWJ_gY
Miscelaneu
http://static.usenix.org/publications/login/2011-02/pdfs/Tange.pdf
http://wiki.scinethpc.ca/wiki/images/7/7b/Tech-talk-gnu-parallel.pdf
http://www.luga.de/Angebote/Vortraege/GNU_Parallel_LIT_2011/GNU_
Parallel_LIT_2011.pdf
http://www.admin-magazine.com/HPC/Articles/
GNU-Parallel-Multicore-at-the-Command-Line-with-GNU-Parallel

39. Adrese web
Comparat, ie cu alte comenzi (xargs, find -exec, make -j etc)
http://www.gnu.org/software/parallel/man.html#summary_table

40. GNU Stow
GNU Stow este un caz particular de manager de pachete.
Instalare:
1 sudo apt−get install stow

41. Instalarea pachetelor
Cum instal˘am un pachet cu ajutorul lui GNU Stow?
1 sudo make install prefix=/usr/local/stow/numepachet
2 cd /usr/local/stow
3 sudo stow numepachet

42. Instalarea pachetelor
Cum instal˘am un pachet cu ajutorul lui GNU Stow?
1 sudo make install prefix=/usr/local/stow/numepachet
2 cd /usr/local/stow
3 sudo stow numepachet
Pachetele care nu folosesc GNU AutoConf pot s˘a nu accepte
parametrul “prefix”, atunci este nevoie de modific˘ari ˆın fis, ierul
Makefile.

43. Eliminarea pachetelor
1 cd /usr/local/stow/
2 sudo stow −D numepachet

44. Exemplu: mai multe versiuni ale unuia s, i aceluias, i pachet
ˆın aceeas, i distribut, ie
Mai multe versiuni ale unuia s, i aceluias, i pachet ˆımpreun˘a.
1 cd numepachet−3.0
2 sudo make install prefix=/usr/local/stow/numepachet−3.0
3 cd numepachet−3.1−beta
4 sudo make install prefix=/usr/local/stow/numepachet−3.1−beta

45. Exemplu: mai multe versiuni ale unuia s, i aceluias, i pachet
ˆın aceeas, i distribut, ie
Putem accesa fis, iere ambelor pachete prin
1 /usr/local/stow/numepachet−3.0/bin/fisierbinar
s, i
2 /usr/local/stow/numepachet−3.1−beta/bin/fisierbinar
Iar dac˘a execut˘am
1 cd /usr/local/stow
2 sudo stow numepachet−3.0
atunci lansˆınd ˆın linia de comand˘a doar fisierbinar vom accesa
/usr/local/stow/numepachet-3.0/bin/fisierbinar.

46. Adrese web
Pagina web
http://www.gnu.org/software/stow/
Manual s, i exemple
http://www.gnu.org/software/stow/manual/