SlideShare a Scribd company logo
1 of 188
Download to read offline
Shooting from the hip.
           Perl at the command




                                                                                                            :)
                                                                                                        @        @
           line for DBAs




                                                                                                             -
                                                                                                            $_
                                                                         #!


                 Giuseppe Maxia
                 MySQL Community Team Lead
                 @datacharmer
       This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Friday, 06 August 2010                                                                                               1
about me -Giuseppe Maxia
        a.k.a. The Data Charmer
        MySQL Community Team Lead
        Long time hacking with MySQL features
        Formerly, database consultant, designer, coder.
        A passion for QA
        An even greater passion for open source
        ... and community
        Passionate blogger
        http://datacharmer.blogspot.com




Friday, 06 August 2010                                     2
Updated presentation slides




    you will find an up to date copy of these slides at:
    http://tinyurl.com/shoot-cl-2010
    http://tinyurl.com/shoot-cl-full-2010




Friday, 06 August 2010                                      3
I love your feedback!
      Tell it to the Twitter community
        #yapc_eu "your feedback here"
      Twitter me
        @datacharmer "your feedback here"
      Blog about it




Friday, 06 August 2010                       4
This is a story about
            languages

Friday, 06 August 2010         5
you got to understand,
                         Bernie, it's not personal.
                           It's just that you are
                                  obsolete
We use
language
to talk to
computers



Friday, 06 August 2010                                6
10 print "hello"
We use                   20 goto 10
languages
to talk to
computers



Friday, 06 August 2010                      7
I am a polyglot



Friday, 06 August 2010                     8
An imperfect
                 polyglot, actually


Friday, 06 August 2010                9
I speak
   Italian




Friday, 06 August 2010   10
I speak
   Italian


       with a clear
        Sardinian
         accent

Friday, 06 August 2010   10
I speak
   English




Friday, 06 August 2010   11
I speak
   English


     with a distinct
     Italian accent

Friday, 06 August 2010   11
I speak
   Perl




Friday, 06 August 2010   12
I speak
   Perl


  with a hint of a C
        accent

Friday, 06 August 2010   12
I speak
   French
   and
   Spanish




Friday, 06 August 2010   13
I speak
   French
   and
   Spanish
   but since I use
  them very little,
 they have melted
  with Italian and
       English
Friday, 06 August 2010   13
I speak
                         #!/bin/bash
   Bash




Friday, 06 August 2010                 14
I speak
                         #!/bin/bash
   Bash


      with a strong
       Perl accent

Friday, 06 August 2010                 14
I speak fluent Italian

                         La donzelletta vien dalla
                         campagna, in sul calar del
                         sole, col suo fascio dell'erba
                         e reca in mano un mazzolino
                         di rose e viole, onde, siccome
                         suole, ornare ella si appresta ...




Friday, 06 August 2010                                        15
However, Sardinian comes handy sometimes




                          Maissavida
                          ti frimisti! (*)




                              (*) May you never stop
Friday, 06 August 2010                                 16
I speak fluent English


                         When in the Course of human events, it becomes
                         necessary for one people to dissolve the political bands
                         which have connected them with another, and to assume
                         among the powers of the earth, the separate and equal
                         station to which the Laws of Nature and of Nature's God
                         entitle them, a decent respect to the opinions of mankind
                         requires that they should declare the causes which impel
                         them to the separation.




Friday, 06 August 2010                                                               17
I speak fluent English


                         When in the Course of human events, it becomes
                         necessary for one people to dissolve the political bands
                         which have connected them with another, and to assume
                         among the powers of the earth, the separate and equal
                         station to which the Laws of Nature and of Nature's God
                         entitle them, a decent respect to the opinions of mankind
                         requires that they should declare the causes which impel
                         them to the separation.



                                                                         hmm.
                                                                         wrong
                                                                         flag

Friday, 06 August 2010                                                               17
I speak fluent English


                         When in the Course of human events, it becomes
                         necessary for one people to dissolve the political bands
                         which have connected them with another, and to assume
                         among the powers of the earth, the separate and equal
                         station to which the Laws of Nature and of Nature's God
                         entitle them, a decent respect to the opinions of mankind
                         requires that they should declare the causes which impel
                         them to the separation.




Friday, 06 August 2010                                                               17
But sometimes Italian is still necessary


                                  barista al fresco pasta
                              maestro        espresso      tempo
                         vendetta        soprano finale          piano
                          dilettante cappuccino           stucco
                         stiletto virtuoso fiasco presto lotto
                              pizza      bravo             lasagna
                                  studio        spaghetti
                                          stanza




Friday, 06 August 2010                                                  18
And sometimes Italian is needed




                           I tuoi perversi
                           defunti!(*)




                           (*) Your wicked ancestors!
Friday, 06 August 2010                                  19
Although Sardinian is still effective here




                              Ancu ti currullidi
                              sa giustizia (*)




                          (*) May the rule of law chase you
Friday, 06 August 2010                                    20
I speak Perl fluently

                         #!/usr/bin/perl
                         use strict;
                         use warnings;
                         use File::Find;

                         while (my $line = <stdin>){
                           # …




Friday, 06 August 2010                                 21
Sometimes, I need help from C

                         use Inline C;
                         print "9 + 16 = ", add(9, 16), "n";

                         __END__
                         __C__
                         int add(int x, int y) {
                               return x + y;
                         }




Friday, 06 August 2010                                          22
Most of my work at the command line is Bash


                         $ for D in $HOME/*/; do du -sh $D;
                         > done > /tmp/occupancy




                                             #!/bin/bash
Friday, 06 August 2010                                        23
And sometimes, a pinch of Perl can help


                  $ perl -pe
                  's{(dd)-(dd)-(dddd)}{$3-$1-$2}g' 
                   < rawfile.txt > betterfile.txt




                                        #!/bin/bash
Friday, 06 August 2010                                         24
And then on top of all this, I need to speak SQL


                  $ mysql -e
                  'SELECT COUNT(*) FROM dbname.tbl_name
                  WHERE col_name = 1 and other_col="A" '




Friday, 06 August 2010                                     25
I wish I could say that my SQL has a Perlish
   accent


                         $ mysql -e
                         'SELECT COUNT(*) FROM dbname.tbl_name
                         WHERE col_name = 1 and other_col="A" '

                            use strict;
                            use DBI;?




Friday, 06 August 2010                                            26
I wish I could say that my SQL has a Perlish
   accent


                         $ mysql -e
                         'SELECT COUNT(*) FROM dbname.tbl_name
                         WHERE col_name = 1 and other_col="A" '



                   but it wouldn't be true




Friday, 06 August 2010                                            27
I could curse in Perl




                           $ perl -pe
                           's{(dd)-(dd)-(ddd
                           d)}{$3-$1-$2}g' 
                            < rawfile.txt >
                           betterfile.txt (*)




   But it would not
   get me far.
                            (*) already translated
Friday, 06 August 2010                                28
Then




Friday, 06 August 2010   29
Then


    I will need to mix languages.




Friday, 06 August 2010              29
Then


    I will need to mix languages.


                         This talk will show exactly that.




Friday, 06 August 2010                                       29
Then


    I will need to mix languages.


                         This talk will show exactly that.


        Mixing languages.




Friday, 06 August 2010                                       29
Then


    I will need to mix languages.


                         This talk will show exactly that.


        Mixing languages.
                                            Yes.



Friday, 06 August 2010                                       29
Then


    I will need to mix languages.


                         This talk will show exactly that.


        Mixing languages.
                                            Yes.

                                 $ at the command line
Friday, 06 August 2010                                       29
What this tutorial is about
    Providing tools and tips for command line users;
    Learn a few tricks to make you achieve more and type
     less.




Friday, 06 August 2010                                  30
What this tutorial is about
    Providing tools and tips for command line users;
    Learn a few tricks to make you achieve more and type
     less.
    Making GUI enthusiasts green with envy;
                                 bar

                          foo
                                         baz




Friday, 06 August 2010                                  30
the command line (for rookies)



                         BASICS
                                     31


Friday, 06 August 2010                    31
The shell
      • It's a command interpreter
      • There are many flavors
           > bash
           > zsh
           > ksh




                                     32


Friday, 06 August 2010                    32
MySQL command line client
      • called "mysql"
           > Takes SQL commands
           > sends them to the server
           > gets results


      • but it can do much more ...




                                        33


Friday, 06 August 2010                       33
other tools
      •   Shell scripts
      •   Aliases
      •   Perl
      •   awk
      •   sort
      •    …. and a lot of Unix little helper



                                                34


Friday, 06 August 2010                               34
other tools
      •   Shell scripts
      •   Aliases
      •   Perl          which could replace everything else, of course
      •   awk
      •   sort
      •    …. and a lot of Unix little helper



                                                                     34


Friday, 06 August 2010                                                    34
in case it wasn't clear (yeah, for
                                  rookies)




                                    35


Friday, 06 August 2010                       35
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are




                                    35


Friday, 06 August 2010                       35
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           > commands (the name of the application to invoke)




                                                                35


Friday, 06 August 2010                                               35
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           > commands (the name of the application to invoke)
           > parameters (optional arguments for the application)




                                                                   35


Friday, 06 August 2010                                                  35
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           > commands (the name of the application to invoke)
           > parameters (optional arguments for the application)
           > and the ENTER key




                                                                   35


Friday, 06 August 2010                                                  35
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           > commands (the name of the application to invoke)
           > parameters (optional arguments for the application)
           > and the ENTER key


                                          Enter

                                                                   35


Friday, 06 August 2010                                                  35
The Unix philosophy (1)
      • One tool for every feature
           > make one tool that does one thing well
           > for many features, make many tools
           > make them work together (see next slide)




                                                        36


Friday, 06 August 2010                                       36
The Unix philosophy (2)
    • To make several tool cooperate:
           > pipe the output of one tool to the input of another one.
           > for more actions, pipe more tools


        $ sort < filename | uniq | nl




                                                                   37


Friday, 06 August 2010                                                  37
Unix pipes




                         38


Friday, 06 August 2010        38
Unix pipes

              application   <   input




                                        38


Friday, 06 August 2010                       38
Unix pipes

              application   <   input

              application   >   output




                                         38


Friday, 06 August 2010                        38
Unix pipes

              application   <   input

              application   >   output


              application   <   input    >   output




                                                      38


Friday, 06 August 2010                                     38
Unix pipes

              application   <   input

              application   >   output


              application   <   input    >   output



              application   |       application


                                                      38


Friday, 06 August 2010                                     38
Unix pipes

              application         <   input

              application         >   output


              application         <   input        >   output



              application         |           application


                         output       input
                                                                38


Friday, 06 August 2010                                               38
Unix pipes = parallel processing




                                   39


Friday, 06 August 2010                  39
Unix pipes = parallel processing

            application   |   application




                                            39


Friday, 06 August 2010                           39
Unix pipes = parallel processing

            application   |   application




                                            39


Friday, 06 August 2010                           39
Unix pipes = parallel processing

            application   |   application

              output




                                            39


Friday, 06 August 2010                           39
Unix pipes = parallel processing

            application   |     application

              output          input




                                              39


Friday, 06 August 2010                             39
Unix pipes = parallel processing

            application   |     application

              output          input




                                              39


Friday, 06 August 2010                             39
Unix pipes = parallel processing

            application   |     application

              output          input




                                              39


Friday, 06 August 2010                             39
Unix pipes = parallel processing

            application   |     application

              output          input




                                              39


Friday, 06 August 2010                             39
Who gets what
      •   shell
      •   application
      •   database server
      •   pager




                            40


Friday, 06 August 2010           40
who gets what


            shell prompt
            $ mysql



                           mysql client




                                          mysql server
                                                         41


Friday, 06 August 2010                                        41
who gets what
                                        any host   mysql server




            shell
                         mysql client




                                                                  42


Friday, 06 August 2010                                                 42
who gets what
$ mysql -u username -ppassword




                         First the application
                         then the server




                                                 43


Friday, 06 August 2010                                43
who gets what
$ mysql -u username -ppassword 
  -e "select VERSION()"



                                 goes to the client, then the
                                 server

            goes to the server




                                                                44


Friday, 06 August 2010                                               44
who gets what
mysql> help
mysql> pager less                             goes to the client only
mysql> select 1;




                         goes to the server




                                                                 45


Friday, 06 August 2010                                                  45
the command line




      readline
                         46


Friday, 06 August 2010        46
readline
      • Common component
           > shell
           > mysql client
           > many CLI applications




                                     47


Friday, 06 August 2010                    47
Main cursor movements
            Ctrl A       Start/End of line
            Ctrl E
            Meta B       Back/Forward one word
            Meta F
                         previous/next command


            Ctrl L       clear screen and take
                         current line on top
                                             48


Friday, 06 August 2010                            48
Changing text
            Meta U            Uppercase word
            Meta L            Lowercase word
            Ctrl D            delete word
            Ctrl K            delete to end of line
            Ctrl -            undo

                         for more tips:
                         $ man readline               49


Friday, 06 August 2010                                     49
Some conventions in this talk
$ command
at the shell

mysql> command
at the mysql prompt




                                    50


Friday, 06 August 2010                   50
the Bash shell



                     #!
                          bash
                             51


Friday, 06 August 2010            51
components
      •   commands
      •   aliases
      •   variables
      •   functions




                         52


Friday, 06 August 2010        52
commands
$ echo "something"
something




                         53


Friday, 06 August 2010        53
aliases
$ alias mydate='mysql -e "select NOW()"'
$ mydate
+---------------------+
| now()               |
+---------------------+
| 2010-04-12 07:14:19 |
+---------------------+




                                 54


Friday, 06 August 2010                 54
aliases
$ alias myself='mysql'
$ myself -e "select NOW()"'
+---------------------+
| now()                |
+---------------------+
| 2010-04-12 07:14:19 |
+---------------------+




                              55


Friday, 06 August 2010             55
variables
$ MYBATCH='mysql -N -B'
$ MYQUERY='select curdate()'
$ $MYBATCH -e "$MYQUERY"
2010-04-12




                               56


Friday, 06 August 2010              56
nesting variables
# t.sh
 CMD="SELECT TABLE_NAME"
 TBL='information_schema.tables'
 WHC1="TABLE_SCHEMA='mysql'"
 WHC2="TABLE_NAME like 'u%' "
 Q1="$CMD FROM $TBL"
 Q2="WHERE $WHC1 AND $WHC2"
 Q="$Q1 $Q2"
 echo $Q
 echo $Q | mysql
                                   57


Friday, 06 August 2010                  57
nesting variables
$ sh t.sh
SELECT TABLE_NAME FROM
information_schema.tables WHERE
TABLE_SCHEMA='mysql' AND TABLE_NAME like
'u%'
TABLE_NAME
user




                                 58


Friday, 06 August 2010                 58
the mysql command line client



                         mysql>

                                    59


Friday, 06 August 2010                   59
SQL commands from a file
   $ mysql < filename.sql
   mysql> source filename.sql

   • executes the commands contained in
     filename.sql
   • commands can be queries understood
     by the server
   • and commands for the client


                                  60


Friday, 06 August 2010                    60
SQL commands at the command line
   $ application | mysql
   $ echo "SELECT 1" | mysql
   $ mysql -e "SELECT 1"

   • executes the commands from standard
     input
   • commands can be queries understood
     by the server
   • and commands for the client

                                  61


Friday, 06 August 2010                     61
batch mode
   $ echo "select curdate()" | mysql
   curdate()
   2010-04-12




                                 62


Friday, 06 August 2010                 62
table mode
   $ echo "select curdate()" | mysql -t
   +------------+
   | curdate() |
   +------------+
   | 2010-04-12 |
   +------------+




                                 63


Friday, 06 August 2010                    63
table mode
   $ mysql -e "select curdate()"
   +------------+
   | curdate() |
   +------------+
   | 2010-04-12 |
   +------------+




                                   64


Friday, 06 August 2010                  64
Forcing batch mode
   mysql -B -e 'select curdate()'
   curdate()
   2010-04-12




                                    65


Friday, 06 August 2010                   65
Forcing batch mode - no header
   mysql -N -B -e 'select curdate()'
   2010-04-12




                                   ANT!
                              PO RT
                         IM

                                          66


Friday, 06 August 2010                         66
Shell + mysql
   $ echo "The current server's date is
   `mysql -N -B -e 'select curdate()'`."

   $ echo "The current server's date is
   $(mysql -N -B -e 'select curdate
   ()')."




                                 67


Friday, 06 August 2010                    67
advanced command line techniques
               ((((((
               (:
           @             @
                -


               $_




                             loops
                                 68


Friday, 06 August 2010                 68
loops basics
   $      for N in 1 2 3
   >      do
   >      echo $N
   >      done
   1
   2
   3



                           69


Friday, 06 August 2010          69
loops basics
   $ for N in 1 2 3 ; do echo $N ; done
   1
   2
   3




                                 70


Friday, 06 August 2010                    70
loops basics
   $ for N in $(seq 1 3) ;
    do echo "this is number $N" ;
    done

   this is number 1
   this is number 2
   this is number 3



                                 71


Friday, 06 August 2010                71
loops from file
   $ head /usr/share/dict/words
   A
   a
   aa
   aal
   aalii
   aam
   Aani
   aardvark
   aardwolf
                                  72


Friday, 06 August 2010                 72
loops from file
   $ for U in 
   $(head /usr/share/dict/words ) ; do
   perl -le 'print uc shift' $U ; done
   A
   A
   AA
   AAL
   AALII
   AAM
   AANI
                                 73


Friday, 06 August 2010                   73
loops from mysql results
   $ mysql -B -N -e 'show tables from
   world'
   City
   Country
   CountryLanguage




                                 74


Friday, 06 August 2010                  74
loops from mysql results
   $ for T in $(mysql -B -N -e 'show
   tables from world') ;
   >    do echo "CREATE TABLE world2.$T
   LIKE world.$T;" ; done

   CREATE TABLE world2.City LIKE
   world.City;
   CREATE TABLE world2.Country LIKE
   world.Country;
   CREATE TABLE world2.CountryLanguage
   LIKE world.CountryLanguage;
                                 75


Friday, 06 August 2010                    75
piping loops back into mysql
   $ for T in $(mysql -B -N -e 'show
   tables from world') ;
   >    do echo "CREATE TABLE world2.$T
   LIKE world.$T;" ; done 
   > | mysql -vv




                                   76


Friday, 06 August 2010                    76
advanced command line techniques



               nested

                                      :)
                                 @         @




                                       -
                                      $_

                         #!
                         #!
                          #!
                          #!



                   loops
                           #!
                            #!




                                 77


Friday, 06 August 2010                         77
doing something for every table
    in every database
   $ mysql -e "show databases"
   +--------------------+
   | Database           |
   +--------------------+
   | information_schema |
   | mysql              |
   | sakila             |
   | world              |
   +--------------------+
                                  78


Friday, 06 August 2010                 78
doing something for every table
    in every database
   $ mysql -e "show tables from world"
   +-----------------+
   | Tables_in_world |
   +-----------------+
   | City            |
   | Country         |
   | CountryLanguage |
   +-----------------+

                                  79


Friday, 06 August 2010                   79
pseudocode
for each database
  do
  for each table
     do
     print the table name




                            80


Friday, 06 August 2010           80
actual code
alias mybatch='mysql -B -N'
for D in $(mybatch -e "SHOW SCHEMAS")
do
   for T in 
   $(mybatch -e "SHOW TABLES FROM $D")
   do
      echo "$D.$T"
   done
done

                                81


Friday, 06 August 2010                   81
advanced command line techniques
                           $HOME




     options
      files                        82


Friday, 06 August 2010                  82
Default option files
/etc/my.cnf
/etc/mysql/my.cnf (Debian, Ubuntu)
$BASEDIR/my.cnf
$HOME/.my.cnf




                               83


Friday, 06 August 2010               83
Custom option files
$ mysql --no-defaults
$ mysql --defaults-file=filename
$ mysql --defaults-extra-file=filename




                               84


Friday, 06 August 2010               84
Option files sections
[mysqld]
# option for server

[client]
# option for any mysql client

[mysql]
# options specific to 'mysql' CLI app


                                85


Friday, 06 August 2010                  85
Option files example

[client]
user=myname
password=mypassword

[mysql]
prompt="mysql (h) {u} [d] > "



                               86


Friday, 06 August 2010              86
Option files: caution
# protect your option files!

$ chmod 600 $HOME/.my.cnf

$ ls -l .my.cnf
-rw------- 1 [...] 10:04 .my.cnf




                               87


Friday, 06 August 2010              87
prompt
mysql> prompt gm (h) {u} [d] >
PROMPT set to 'gm (h) {u} [d] > '
gm (localhost) {root} [(none)] > use test
Database changed
gm (localhost) {root} [test] >

# h -> host
# u -> user
# d -> database

                                 88


Friday, 06 August 2010                 88
advanced command line techniques
            ((((((
            (:

        @            @
             -




            #!




                         aliases
                               89


Friday, 06 August 2010                 89
An useful alias
$ alias mysql_as_root=
'mysql --
defaults_file=~/.my_as_root.cnf'

mysql_as_root -e "select user()"
+----------------+
| user()         |
+----------------+
| root@localhost |
+----------------+
                               90


Friday, 06 August 2010              90
An useful alias
#.my_as_root.cnf'
[client]
user=root
password=your_mysql_root_password

[mysql]
prompt="### [h] {u} (d) > "



                                 91


Friday, 06 August 2010                91
Another useful alias
alias mybatch='mysql -B -N '


mybatch -e "select version()"
5.1.45-log




                                92


Friday, 06 August 2010               92
( )
    advanced command line techniques
                    ((((((
                    (:
                @            @
                    -



                    #!




   functions
                                 93


Friday, 06 August 2010                 93
functions
      • like aliases, but with parameters




                                            94


Friday, 06 August 2010                           94
sample function
function grant_new {
    mysql_as_root -e 
    "GRANT $1 on $2 to $3 identified
by "$4" "
}


$ grant_new SELECT "world.*" john 
  john_secret

                               95


Friday, 06 August 2010                 95
sample function
$ mysql_as_root 
  -e 'show grants for johnG'
*** 1. row ***
Grants for john@%: GRANT USAGE ON *.*
TO 'john'@'%' IDENTIFIED BY PASSWORD
'*73A88CE13374128844E3ABB500F439A9C02C
53A3'
*** 2. row ***
Grants for john@%: GRANT SELECT ON
`world`.* TO 'john'@'%'

                               96


Friday, 06 August 2010               96
advanced command line techniques



              $_         getting
                          query
                         details
                               97


Friday, 06 August 2010                 97
which query is being executed?
# wrong.sql
# this script has one error
use test ;
drop table if exists t1 ;
create table t1 (id int);
insert into t1 values (a);




                                 98


Friday, 06 August 2010                98
which query is being executed?
$ mysql < wrong.sql
ERROR 1054 (42S22) at line 4: Unknown
column 'a' in 'field list'


# WHERE IS THE ERROR?




                                 99


Friday, 06 August 2010                  99
which query is being executed?
$ mysql -v < wrong.sql
--------------
drop table if exists t1
--------------
--------------
create table t1 (id int)
--------------
--------------
insert into t1 values (a)
--------------
ERROR 1054 (42S22) at line 4: Unknown column
'a' in 'field list'
                                      100


Friday, 06 August 2010                         100
getting execution times
$ mysql -vv -e 
'select rating,count(*) from sakila.film
group by rating'
+--------+----------+
| rating | count(*) |
+--------+----------+
| G      |       178 |
| PG     |       194 |
| PG-13 |        223 |
| R      |       195 |
| NC-17 |        210 |
+--------+----------+
5 rows in set (0.00 sec)            101


Friday, 06 August 2010                     101
getting execution times (batch mode)
$echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vv
--------------
select rating,count(*) from sakila.film group by
rating
--------------

rating count(*)
G 178
PG 194
PG-13 223
R 195
NC-17 210
5 rows in set

                                             102


Friday, 06 August 2010                                 102
getting execution times (batch mode)
$ echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vvv
--------------
select rating,count(*) from sakila.film group by
rating
--------------
+--------+----------+
| rating | count(*) |
+--------+----------+
| G      |      178 |
| PG     |      194 |
| PG-13 |       223 |
| R      |      195 |
| NC-17 |       210 |
+--------+----------+
                                             103
5 rows in set (0.00 sec)
Friday, 06 August 2010                                  103
advanced command line techniques



                         getting
        _$

                         distinct
                         results
                               104


Friday, 06 August 2010                 104
getting a single value
$ mysql -e "show variables like 'port'"
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| port          | 3306 |
+---------------+-------+

# Here, we only want the port number




                                       105


Friday, 06 August 2010                       105
getting a single value
$ mysql -NB -e "show variables like 'port'"
 port           3306


# no header, batch mode
# still two values




                                    106


Friday, 06 August 2010                    106
getting a single value
$ mysql -NB 
  -e "show variables like 'port'" 
  | awk '{print $2}'

3306

$ mysql -NB 
  -e "show variables like 'port'" 
 | perl -lane 'print $F[1]'

3306
# filtering the wanted value
                                      107


Friday, 06 August 2010                      107
getting a single value: is replication
    working?
$ mysql 
  -e "show slave status G" 
 | grep -i slave_io_running

                         Slave_IO_Running: Yes


# too many things.
# We only want a 'yes' or 'no'

                                                 108


Friday, 06 August 2010                                 108
getting a single value: is replication
    working?
$ mysql 
  -e "show slave status G" 
  | grep -i "slave_io_running" 
  | perl -lane 'print $F[1]'

Yes


# yes!

                                     109


Friday, 06 August 2010                       109
What's perl -lane?




                         110


Friday, 06 August 2010         110
What's perl -lane?   perl -lane
                         must
                         turn perl




                             110


Friday, 06 August 2010             110
Perl command line must-know facts (1)
perl -e 'print "hellon"'
# executes perl code




                              111


Friday, 06 August 2010              111
Perl command line must-know facts (2)
perl -ne 'print if /^a/'
# starts a loop from standard input
# and executes the perl code for each
# iteration

perl -MO=Deparse -ne 'print if /^a/'
LINE: while (defined($_ = <ARGV>)) {
    print $_ if /^a/;
}

                               112


Friday, 06 August 2010                  112
Perl command line must-know facts (3)
perl -pe 's/(^aw+)/$1/'
# like -ne, but prints each line

$ perl -MO=Deparse -pe 's/(^aw+)/$1/'
LINE: while (defined($_ = <ARGV>)) {
    s/(^aw+)/$1/;
}
continue {
    print $_;
}
                               113


Friday, 06 August 2010               113
Perl command line must-know facts (4)
perl -le 'print "hello"'
# strips newlines from input and adds
# them to output

$ perl -e 'print "hello"'
hello$

$ perl -le 'print "hello"'
hello
$
                               114


Friday, 06 August 2010                  114
Perl command line must-know facts (5)
perl -lane 'print $F[0]'
# splits input into @F array

$ echo "a b c" | 
   perl -lane 'print $F[1]'
b

$ echo "a b c" | 
  perl -lane 'print $F[2]'
c
                               115


Friday, 06 August 2010               115
Perl command line must-know facts (6)
Perl and shell contention for $




 #!/bin/bash

                         Take the $$$ and run
                                            116


Friday, 06 August 2010                            116
Beware of $VARIABLES on the
command line
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.




                               117


Friday, 06 August 2010                117
double quotes
Beware of $VARIABLES on theconsumed by the
                           allow the shell to
                           shell, before Perl
                           interpolate
command line               can see it
                           variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.




                                   117


Friday, 06 August 2010                          117
double quotes
Beware of $VARIABLES on theconsumed by the
                           allow the shell to
                           shell, before Perl
                           interpolate
command line               can see it
                           variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
                            consumed by the
                            shell, before Perl
                            can see it




                                   117


Friday, 06 August 2010                           117
double quotes
Beware of $VARIABLES on the   consumed by the
                              allow the shell to
                              shell, before Perl
                              interpolate
command line                  can see it
                              variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
                               consumed by the
                               shell, before Perl
$ perl -le '$x=1; print $x'    can see it
1


                                      117


Friday, 06 August 2010                              117
double quotes
Beware of $VARIABLES on the   consumed by the
                              allow the shell to
                              shell, before Perl
                              interpolate
command line                  can see it
                              variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
                               consumed by the
                               shell, before Perl
$ perl -le '$x=1; print $x'    can see it
1
                              single quotes the
                              consumed by
                              prevent variable
                              shell, before Perl
                              consumption by
                              can shell it
                              the see117
Friday, 06 August 2010                              117
advanced command line techniques


         combining
          column
           names
                               118


Friday, 06 August 2010                 118
combining column names
$ mysql -N -e 'desc world.City'
+-------------+----------+----+-----+
|          ID | int(11) | NO | PRI |
|        Name | char(35) | NO |     |
| CountryCode | char(3) | NO |      |
|    District | char(20) | NO |     |
| Population | int(11) | NO |       |
+-------------+----------+----+-----+



                                   119


Friday, 06 August 2010                   119
combining column names
#     Expected result:
#     SELECT CONCAT_WS('|',
#     ID,Name,CountryCode,District,
#     Population) from City




                                      120


Friday, 06 August 2010                      120
combining column names (I)
echo "select concat_ws('|',
  $(mysql -N -e 'desc world.City' | 
   perl -lane 'push @f, $F[0];' 
        -e 'END{print join ",", @f}'))
from world.City"
select concat_ws('|',
ID,Name,CountryCode,District,Populatio
n) from world.City


                                 121


Friday, 06 August 2010                 121
combining column names (II)
echo "select concat_ws('|', $(
  mysql -NB -e 'select group_concat
(column_name) from
information_schema.columns where
table_name="City" and
table_schema="world"')) from
world.City"
select concat_ws('|',
ID,Name,CountryCode,District,Populatio
n) from world.City
                                  122


Friday, 06 August 2010                  122
combining column names (IIa)
echo "select concat_ws('|', $(
  mysql -NB -e 'select group_concat
(column_name) from
information_schema.columns where
table_name="Country" and
table_schema="world"')) from
world.Country"
select concat_ws('|',
Code,Name,Continent,Region,SurfaceArea,Inde
pYear,Population,LifeExpectancy,GNP,GNPOld,
LocalName,GovernmentForm,HeadOfState,Capita
l,Code2) from world.Country         123


Friday, 06 August 2010                    123
piping the result back to mysql
echo "select concat_ws('|', $(
  mysql -NB -e 'select group_concat
(column_name) from
information_schema.columns where
table_name="Country" and
table_schema="world"')) as line from
world.Country" | mysql | head -n 4
line
AFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919|
22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate|
Mohammad Omar|1|AF
NLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000|
78.3|371362.00|360478.00|Nederland|Constitutional Monarchy|
Beatrix|5|NL
ANT|Netherlands Antilles|North America|Caribbean|800.00|217000|
74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of
                                                      124
The Netherlands|Beatrix|33|AN
Friday, 06 August 2010                                            124
advanced command line techniques


                 $_

                         dumping
                          data
                                125


Friday, 06 August 2010                  125
advanced command line techniques


                 $_

                         dumping
                          data
                                125


Friday, 06 August 2010                  125
dumping a tab delimited data set
# the naive way
$ mysql 
  -e "SELECT *
  into outfile '/tmp/city_data'
from world.City"

# it works only if:
# - /tmp/city_data does not exist
# - /tmp is world writable

                                  126


Friday, 06 August 2010                  126
dumping a tab delimited data set
$ mysql -B -N 
  -e "SELECT * from world.City" 
  > /tmp/city_data

# it works always

# if you omit the "-N" it also
# includes a header


                                  127


Friday, 06 August 2010                  127
dumping a CSV
$ mysql -B 
  -e "SELECT * from world.City" 
 | perl -F"t" -lane 
  'print join ",",
   map {s/"/""/g;
      /^[d.]+$/ ? $_ : qq("$_")}
   @F ' 
  > /tmp/city_data


                               128


Friday, 06 August 2010               128
dumping a CSV
$ mysql -B 
  -e "SELECT * from world.City" 
 | perl -F"t" -lane 
  'print join ",",    splits the line by tabs, instead
   map {s/"/""/g;     of the default whitespace
      /^[d.]+$/ ? $_ : qq("$_")}
   @F ' 
  > /tmp/city_data


                                               128


Friday, 06 August 2010                                   128
advanced command line techniques




      script
        it
                               129


Friday, 06 August 2010                 129
use the mysql.sh script
            The companion code for this tutorial
            http://tinyurl.com/cli-tut-code-2010
            It will save you much typing
            But use it at your own risk!




                                            130


Friday, 06 August 2010                              130
Usage
      • download the script
      • load it

         $ source mysql.sh




                              131


Friday, 06 August 2010              131
mysql.sh functions
      •   my_batch "query"
      •   returns the result of the query.
      •   Without header
      •   Without box

    $      my_batch "SELECT * FROM test.t1"
    1      aaa
    2      bbb
    3      ccc
                                             132


Friday, 06 August 2010                             132
mysql.sh functions
      • my_value "what"
      • returns a single value

      • e.g.
      $ my_value "version()"
      5.1.49

      $ my_value "@@server_id"
      1
                                 133


Friday, 06 August 2010                 133
mysql.sh functions
      • my_var "what"
      • returns the value of a given variable

      • e.g.
      $ my_var port
      3306

      $ my_var datadir
      /usr/local/mysql/data
                                                134


Friday, 06 August 2010                                134
mysql.sh functions
      • my_status "what"
      • returns the value of a given status item

      • e.g.
      $ my_status com_select
      2653

      $ my_status connections
      452
                                                   135


Friday, 06 August 2010                                   135
mysql.sh functions
      • my_dump_dataset "query" [header]
      • returns the values of a given query separated by tabs
      • if a header is requested, it will be part of the result
      • e.g.
      $ my_dump_dataset "select * from
      world.City" | head -n 5
      1 Kabul AFG Kabol 1780000
      2 Qandahar AFG Qandahar 237500
      3 Herat AFG Herat 186800
      4 Mazar-e-Sharif AFG Balkh 127800
                                                            136
      5 Amsterdam NLD Noord-Holland 731200
Friday, 06 August 2010                                            136
mysql.sh functions
      • my_csv "query" [separator]
      • returns the values of a given query separated by
        commas, or by a given separator
      • e.g.
$ my_csv 'select * from world.City' |
head -n 3
"ID","Name","CountryCode","District","
Population"
1,"Kabul","AFG","Kabol",1780000
2,"Qandahar","AFG","Qandahar",237500
                                                       137


Friday, 06 August 2010                                       137
inside the command line client

     handling


                                           :-)
                                       @         @


                                           $_



                         output
                                     138


Friday, 06 August 2010                               138
pager
mysql> SELECT * FROM world.City;

rec …
rec …
rec …

4000 + records
unreadable


                               139


Friday, 06 August 2010               139
pager
mysql> pager less
PAGER set to 'less'
mysql> SELECT * FROM world.City;

# [ records in a manageable window ]

mysql> nopager
PAGER set to stdout

# back to normal
                               140


Friday, 06 August 2010                 140
More pager tricks
mysql> pager vim -
# like 'less', but you can also edit
# the results and eventually save
# them




                               141


Friday, 06 August 2010                 141
More pager tricks
mysql> pager md5sum
PAGER set to 'md5sum'
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543          -
4079 rows in set (0.01 sec)

#      produce a CRC for the result set
#      instead of displaying it.
#      Useful for data comparison.
#      (Thanks to Baron Schwartz
                                    142


Friday, 06 August 2010                        142
Comparing data sets with md5sum
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543   -

mysql> select * from City order by ID;
5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by name;
5ab980f5b69a6e65d3b8cd5a95ec94e0 -

mysql> select * from City order by
District;
2bf37cbe91b3ffeaebc8c4b6084b8784 -     143


Friday, 06 August 2010                       143
Comparing data sets with md5sum
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543   -

mysql> select * from City order by ID;
5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by name;
5ab980f5b69a6e65d3b8cd5a95ec94e0 -

mysql> select * from City order by
District;
2bf37cbe91b3ffeaebc8c4b6084b8784 -     143


Friday, 06 August 2010                       143
More pager tricks
mysql>                   pager cat > /tmp/t1.txt
mysql>                   select "one" union select
"two";
mysql>                   pager cat > /tmp/t2.txt
mysql>                   select "one" union select
"TWO";
mysql>                   nopager
mysql>                   ! vimdiff -o /tmp/t[12].txt

# see the difference between two
# datasets
                                                 144


Friday, 06 August 2010                                  144
145


Friday, 06 August 2010         145
logging sessions
mysql> tee mylog.txt

#      logs everything that you see on
#      the screen.
#      your commands, the results,
#      warnings, error messages, and so on.

mysql> notee

# back to normal
                                    146


Friday, 06 August 2010                    146
Updated presentation slides




    you will find an up to date copy of these slides at:
    http://tinyurl.com/shoot-cl-2010
    http://tinyurl.com/shoot-cl-full-2010




Friday, 06 August 2010                                      147
read more
    http://datacharmer.blogspot.com




Friday, 06 August 2010                 148
THANKS!
             ?
                                  Question
                                    time
                                Comments on Twitter:
                                @datacharmer
      This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Friday, 06 August 2010                                                                                 149

More Related Content

More from Giuseppe Maxia

MySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployerMySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployerGiuseppe Maxia
 
Dbdeployer, the universal installer
Dbdeployer, the universal installerDbdeployer, the universal installer
Dbdeployer, the universal installerGiuseppe Maxia
 
Test complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployerTest complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployerGiuseppe Maxia
 
A quick tour of Mysql 8 roles
A quick tour of Mysql 8 rolesA quick tour of Mysql 8 roles
A quick tour of Mysql 8 rolesGiuseppe Maxia
 
Synchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDBSynchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDBGiuseppe Maxia
 
Juggle your data with Tungsten Replicator
Juggle your data with Tungsten ReplicatorJuggle your data with Tungsten Replicator
Juggle your data with Tungsten ReplicatorGiuseppe Maxia
 
Tungsten Replicator tutorial
Tungsten Replicator tutorialTungsten Replicator tutorial
Tungsten Replicator tutorialGiuseppe Maxia
 
Preventing multi master conflicts with tungsten
Preventing multi master conflicts with tungstenPreventing multi master conflicts with tungsten
Preventing multi master conflicts with tungstenGiuseppe Maxia
 
MySQL high availability power and usability
MySQL high availability power and usabilityMySQL high availability power and usability
MySQL high availability power and usabilityGiuseppe Maxia
 
Solving MySQL replication problems with Tungsten
Solving MySQL replication problems with TungstenSolving MySQL replication problems with Tungsten
Solving MySQL replication problems with TungstenGiuseppe Maxia
 
State of the art of MySQL replication and clustering
State of the art of MySQL replication and clusteringState of the art of MySQL replication and clustering
State of the art of MySQL replication and clusteringGiuseppe Maxia
 
Testing mysql creatively in a sandbox
Testing mysql creatively in a sandboxTesting mysql creatively in a sandbox
Testing mysql creatively in a sandboxGiuseppe Maxia
 
Mysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replicationMysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replicationGiuseppe Maxia
 

More from Giuseppe Maxia (20)

MySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployerMySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployer
 
Test like a_boss
Test like a_bossTest like a_boss
Test like a_boss
 
Dbdeployer, the universal installer
Dbdeployer, the universal installerDbdeployer, the universal installer
Dbdeployer, the universal installer
 
Test complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployerTest complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployer
 
Dbdeployer
DbdeployerDbdeployer
Dbdeployer
 
Dbdeployer
DbdeployerDbdeployer
Dbdeployer
 
A quick tour of Mysql 8 roles
A quick tour of Mysql 8 rolesA quick tour of Mysql 8 roles
A quick tour of Mysql 8 roles
 
MySQL document_store
MySQL document_storeMySQL document_store
MySQL document_store
 
Replication skeptic
Replication skepticReplication skeptic
Replication skeptic
 
Synchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDBSynchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDB
 
Juggle your data with Tungsten Replicator
Juggle your data with Tungsten ReplicatorJuggle your data with Tungsten Replicator
Juggle your data with Tungsten Replicator
 
MySQL in your laptop
MySQL in your laptopMySQL in your laptop
MySQL in your laptop
 
Script it
Script itScript it
Script it
 
Tungsten Replicator tutorial
Tungsten Replicator tutorialTungsten Replicator tutorial
Tungsten Replicator tutorial
 
Preventing multi master conflicts with tungsten
Preventing multi master conflicts with tungstenPreventing multi master conflicts with tungsten
Preventing multi master conflicts with tungsten
 
MySQL high availability power and usability
MySQL high availability power and usabilityMySQL high availability power and usability
MySQL high availability power and usability
 
Solving MySQL replication problems with Tungsten
Solving MySQL replication problems with TungstenSolving MySQL replication problems with Tungsten
Solving MySQL replication problems with Tungsten
 
State of the art of MySQL replication and clustering
State of the art of MySQL replication and clusteringState of the art of MySQL replication and clustering
State of the art of MySQL replication and clustering
 
Testing mysql creatively in a sandbox
Testing mysql creatively in a sandboxTesting mysql creatively in a sandbox
Testing mysql creatively in a sandbox
 
Mysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replicationMysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replication
 

Recently uploaded

"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 

Recently uploaded (20)

"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector Databases
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 

Shooting from the hip - Perl at the command line for DBAs

  • 1. Shooting from the hip. Perl at the command :) @ @ line for DBAs - $_ #! Giuseppe Maxia MySQL Community Team Lead @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 1
  • 2. about me -Giuseppe Maxia  a.k.a. The Data Charmer  MySQL Community Team Lead  Long time hacking with MySQL features  Formerly, database consultant, designer, coder.  A passion for QA  An even greater passion for open source  ... and community  Passionate blogger  http://datacharmer.blogspot.com Friday, 06 August 2010 2
  • 3. Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 3
  • 4. I love your feedback!  Tell it to the Twitter community  #yapc_eu "your feedback here"  Twitter me  @datacharmer "your feedback here"  Blog about it Friday, 06 August 2010 4
  • 5. This is a story about languages Friday, 06 August 2010 5
  • 6. you got to understand, Bernie, it's not personal. It's just that you are obsolete We use language to talk to computers Friday, 06 August 2010 6
  • 7. 10 print "hello" We use 20 goto 10 languages to talk to computers Friday, 06 August 2010 7
  • 8. I am a polyglot Friday, 06 August 2010 8
  • 9. An imperfect polyglot, actually Friday, 06 August 2010 9
  • 10. I speak Italian Friday, 06 August 2010 10
  • 11. I speak Italian with a clear Sardinian accent Friday, 06 August 2010 10
  • 12. I speak English Friday, 06 August 2010 11
  • 13. I speak English with a distinct Italian accent Friday, 06 August 2010 11
  • 14. I speak Perl Friday, 06 August 2010 12
  • 15. I speak Perl with a hint of a C accent Friday, 06 August 2010 12
  • 16. I speak French and Spanish Friday, 06 August 2010 13
  • 17. I speak French and Spanish but since I use them very little, they have melted with Italian and English Friday, 06 August 2010 13
  • 18. I speak #!/bin/bash Bash Friday, 06 August 2010 14
  • 19. I speak #!/bin/bash Bash with a strong Perl accent Friday, 06 August 2010 14
  • 20. I speak fluent Italian La donzelletta vien dalla campagna, in sul calar del sole, col suo fascio dell'erba e reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ... Friday, 06 August 2010 15
  • 21. However, Sardinian comes handy sometimes Maissavida ti frimisti! (*) (*) May you never stop Friday, 06 August 2010 16
  • 22. I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17
  • 23. I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. hmm. wrong flag Friday, 06 August 2010 17
  • 24. I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17
  • 25. But sometimes Italian is still necessary barista al fresco pasta maestro espresso tempo vendetta soprano finale piano dilettante cappuccino stucco stiletto virtuoso fiasco presto lotto pizza bravo lasagna studio spaghetti stanza Friday, 06 August 2010 18
  • 26. And sometimes Italian is needed I tuoi perversi defunti!(*) (*) Your wicked ancestors! Friday, 06 August 2010 19
  • 27. Although Sardinian is still effective here Ancu ti currullidi sa giustizia (*) (*) May the rule of law chase you Friday, 06 August 2010 20
  • 28. I speak Perl fluently #!/usr/bin/perl use strict; use warnings; use File::Find; while (my $line = <stdin>){ # … Friday, 06 August 2010 21
  • 29. Sometimes, I need help from C use Inline C; print "9 + 16 = ", add(9, 16), "n"; __END__ __C__ int add(int x, int y) { return x + y; } Friday, 06 August 2010 22
  • 30. Most of my work at the command line is Bash $ for D in $HOME/*/; do du -sh $D; > done > /tmp/occupancy #!/bin/bash Friday, 06 August 2010 23
  • 31. And sometimes, a pinch of Perl can help $ perl -pe 's{(dd)-(dd)-(dddd)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt #!/bin/bash Friday, 06 August 2010 24
  • 32. And then on top of all this, I need to speak SQL $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' Friday, 06 August 2010 25
  • 33. I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' use strict; use DBI;? Friday, 06 August 2010 26
  • 34. I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' but it wouldn't be true Friday, 06 August 2010 27
  • 35. I could curse in Perl $ perl -pe 's{(dd)-(dd)-(ddd d)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt (*) But it would not get me far. (*) already translated Friday, 06 August 2010 28
  • 37. Then I will need to mix languages. Friday, 06 August 2010 29
  • 38. Then I will need to mix languages. This talk will show exactly that. Friday, 06 August 2010 29
  • 39. Then I will need to mix languages. This talk will show exactly that. Mixing languages. Friday, 06 August 2010 29
  • 40. Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. Friday, 06 August 2010 29
  • 41. Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. $ at the command line Friday, 06 August 2010 29
  • 42. What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less. Friday, 06 August 2010 30
  • 43. What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less.  Making GUI enthusiasts green with envy; bar foo baz Friday, 06 August 2010 30
  • 44. the command line (for rookies) BASICS 31 Friday, 06 August 2010 31
  • 45. The shell • It's a command interpreter • There are many flavors > bash > zsh > ksh 32 Friday, 06 August 2010 32
  • 46. MySQL command line client • called "mysql" > Takes SQL commands > sends them to the server > gets results • but it can do much more ... 33 Friday, 06 August 2010 33
  • 47. other tools • Shell scripts • Aliases • Perl • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34
  • 48. other tools • Shell scripts • Aliases • Perl which could replace everything else, of course • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34
  • 49. in case it wasn't clear (yeah, for rookies) 35 Friday, 06 August 2010 35
  • 50. in case it wasn't clear (yeah, for rookies) • command line actions are 35 Friday, 06 August 2010 35
  • 51. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) 35 Friday, 06 August 2010 35
  • 52. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) 35 Friday, 06 August 2010 35
  • 53. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key 35 Friday, 06 August 2010 35
  • 54. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key Enter 35 Friday, 06 August 2010 35
  • 55. The Unix philosophy (1) • One tool for every feature > make one tool that does one thing well > for many features, make many tools > make them work together (see next slide) 36 Friday, 06 August 2010 36
  • 56. The Unix philosophy (2) • To make several tool cooperate: > pipe the output of one tool to the input of another one. > for more actions, pipe more tools $ sort < filename | uniq | nl 37 Friday, 06 August 2010 37
  • 57. Unix pipes 38 Friday, 06 August 2010 38
  • 58. Unix pipes application < input 38 Friday, 06 August 2010 38
  • 59. Unix pipes application < input application > output 38 Friday, 06 August 2010 38
  • 60. Unix pipes application < input application > output application < input > output 38 Friday, 06 August 2010 38
  • 61. Unix pipes application < input application > output application < input > output application | application 38 Friday, 06 August 2010 38
  • 62. Unix pipes application < input application > output application < input > output application | application output input 38 Friday, 06 August 2010 38
  • 63. Unix pipes = parallel processing 39 Friday, 06 August 2010 39
  • 64. Unix pipes = parallel processing application | application 39 Friday, 06 August 2010 39
  • 65. Unix pipes = parallel processing application | application 39 Friday, 06 August 2010 39
  • 66. Unix pipes = parallel processing application | application output 39 Friday, 06 August 2010 39
  • 67. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  • 68. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  • 69. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  • 70. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  • 71. Who gets what • shell • application • database server • pager 40 Friday, 06 August 2010 40
  • 72. who gets what shell prompt $ mysql mysql client mysql server 41 Friday, 06 August 2010 41
  • 73. who gets what any host mysql server shell mysql client 42 Friday, 06 August 2010 42
  • 74. who gets what $ mysql -u username -ppassword First the application then the server 43 Friday, 06 August 2010 43
  • 75. who gets what $ mysql -u username -ppassword -e "select VERSION()" goes to the client, then the server goes to the server 44 Friday, 06 August 2010 44
  • 76. who gets what mysql> help mysql> pager less goes to the client only mysql> select 1; goes to the server 45 Friday, 06 August 2010 45
  • 77. the command line readline 46 Friday, 06 August 2010 46
  • 78. readline • Common component > shell > mysql client > many CLI applications 47 Friday, 06 August 2010 47
  • 79. Main cursor movements Ctrl A Start/End of line Ctrl E Meta B Back/Forward one word Meta F previous/next command Ctrl L clear screen and take current line on top 48 Friday, 06 August 2010 48
  • 80. Changing text Meta U Uppercase word Meta L Lowercase word Ctrl D delete word Ctrl K delete to end of line Ctrl - undo for more tips: $ man readline 49 Friday, 06 August 2010 49
  • 81. Some conventions in this talk $ command at the shell mysql> command at the mysql prompt 50 Friday, 06 August 2010 50
  • 82. the Bash shell #! bash 51 Friday, 06 August 2010 51
  • 83. components • commands • aliases • variables • functions 52 Friday, 06 August 2010 52
  • 84. commands $ echo "something" something 53 Friday, 06 August 2010 53
  • 85. aliases $ alias mydate='mysql -e "select NOW()"' $ mydate +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 54 Friday, 06 August 2010 54
  • 86. aliases $ alias myself='mysql' $ myself -e "select NOW()"' +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 55 Friday, 06 August 2010 55
  • 87. variables $ MYBATCH='mysql -N -B' $ MYQUERY='select curdate()' $ $MYBATCH -e "$MYQUERY" 2010-04-12 56 Friday, 06 August 2010 56
  • 88. nesting variables # t.sh CMD="SELECT TABLE_NAME" TBL='information_schema.tables' WHC1="TABLE_SCHEMA='mysql'" WHC2="TABLE_NAME like 'u%' " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql 57 Friday, 06 August 2010 57
  • 89. nesting variables $ sh t.sh SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='mysql' AND TABLE_NAME like 'u%' TABLE_NAME user 58 Friday, 06 August 2010 58
  • 90. the mysql command line client mysql> 59 Friday, 06 August 2010 59
  • 91. SQL commands from a file $ mysql < filename.sql mysql> source filename.sql • executes the commands contained in filename.sql • commands can be queries understood by the server • and commands for the client 60 Friday, 06 August 2010 60
  • 92. SQL commands at the command line $ application | mysql $ echo "SELECT 1" | mysql $ mysql -e "SELECT 1" • executes the commands from standard input • commands can be queries understood by the server • and commands for the client 61 Friday, 06 August 2010 61
  • 93. batch mode $ echo "select curdate()" | mysql curdate() 2010-04-12 62 Friday, 06 August 2010 62
  • 94. table mode $ echo "select curdate()" | mysql -t +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 63 Friday, 06 August 2010 63
  • 95. table mode $ mysql -e "select curdate()" +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 64 Friday, 06 August 2010 64
  • 96. Forcing batch mode mysql -B -e 'select curdate()' curdate() 2010-04-12 65 Friday, 06 August 2010 65
  • 97. Forcing batch mode - no header mysql -N -B -e 'select curdate()' 2010-04-12 ANT! PO RT IM 66 Friday, 06 August 2010 66
  • 98. Shell + mysql $ echo "The current server's date is `mysql -N -B -e 'select curdate()'`." $ echo "The current server's date is $(mysql -N -B -e 'select curdate ()')." 67 Friday, 06 August 2010 67
  • 99. advanced command line techniques (((((( (: @ @ - $_ loops 68 Friday, 06 August 2010 68
  • 100. loops basics $ for N in 1 2 3 > do > echo $N > done 1 2 3 69 Friday, 06 August 2010 69
  • 101. loops basics $ for N in 1 2 3 ; do echo $N ; done 1 2 3 70 Friday, 06 August 2010 70
  • 102. loops basics $ for N in $(seq 1 3) ; do echo "this is number $N" ; done this is number 1 this is number 2 this is number 3 71 Friday, 06 August 2010 71
  • 103. loops from file $ head /usr/share/dict/words A a aa aal aalii aam Aani aardvark aardwolf 72 Friday, 06 August 2010 72
  • 104. loops from file $ for U in $(head /usr/share/dict/words ) ; do perl -le 'print uc shift' $U ; done A A AA AAL AALII AAM AANI 73 Friday, 06 August 2010 73
  • 105. loops from mysql results $ mysql -B -N -e 'show tables from world' City Country CountryLanguage 74 Friday, 06 August 2010 74
  • 106. loops from mysql results $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done CREATE TABLE world2.City LIKE world.City; CREATE TABLE world2.Country LIKE world.Country; CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage; 75 Friday, 06 August 2010 75
  • 107. piping loops back into mysql $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done > | mysql -vv 76 Friday, 06 August 2010 76
  • 108. advanced command line techniques nested :) @ @ - $_ #! #! #! #! loops #! #! 77 Friday, 06 August 2010 77
  • 109. doing something for every table in every database $ mysql -e "show databases" +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | sakila | | world | +--------------------+ 78 Friday, 06 August 2010 78
  • 110. doing something for every table in every database $ mysql -e "show tables from world" +-----------------+ | Tables_in_world | +-----------------+ | City | | Country | | CountryLanguage | +-----------------+ 79 Friday, 06 August 2010 79
  • 111. pseudocode for each database do for each table do print the table name 80 Friday, 06 August 2010 80
  • 112. actual code alias mybatch='mysql -B -N' for D in $(mybatch -e "SHOW SCHEMAS") do for T in $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" done done 81 Friday, 06 August 2010 81
  • 113. advanced command line techniques $HOME options files 82 Friday, 06 August 2010 82
  • 114. Default option files /etc/my.cnf /etc/mysql/my.cnf (Debian, Ubuntu) $BASEDIR/my.cnf $HOME/.my.cnf 83 Friday, 06 August 2010 83
  • 115. Custom option files $ mysql --no-defaults $ mysql --defaults-file=filename $ mysql --defaults-extra-file=filename 84 Friday, 06 August 2010 84
  • 116. Option files sections [mysqld] # option for server [client] # option for any mysql client [mysql] # options specific to 'mysql' CLI app 85 Friday, 06 August 2010 85
  • 118. Option files: caution # protect your option files! $ chmod 600 $HOME/.my.cnf $ ls -l .my.cnf -rw------- 1 [...] 10:04 .my.cnf 87 Friday, 06 August 2010 87
  • 119. prompt mysql> prompt gm (h) {u} [d] > PROMPT set to 'gm (h) {u} [d] > ' gm (localhost) {root} [(none)] > use test Database changed gm (localhost) {root} [test] > # h -> host # u -> user # d -> database 88 Friday, 06 August 2010 88
  • 120. advanced command line techniques (((((( (: @ @ - #! aliases 89 Friday, 06 August 2010 89
  • 121. An useful alias $ alias mysql_as_root= 'mysql -- defaults_file=~/.my_as_root.cnf' mysql_as_root -e "select user()" +----------------+ | user() | +----------------+ | root@localhost | +----------------+ 90 Friday, 06 August 2010 90
  • 123. Another useful alias alias mybatch='mysql -B -N ' mybatch -e "select version()" 5.1.45-log 92 Friday, 06 August 2010 92
  • 124. ( ) advanced command line techniques (((((( (: @ @ - #! functions 93 Friday, 06 August 2010 93
  • 125. functions • like aliases, but with parameters 94 Friday, 06 August 2010 94
  • 126. sample function function grant_new { mysql_as_root -e "GRANT $1 on $2 to $3 identified by "$4" " } $ grant_new SELECT "world.*" john john_secret 95 Friday, 06 August 2010 95
  • 127. sample function $ mysql_as_root -e 'show grants for johnG' *** 1. row *** Grants for john@%: GRANT USAGE ON *.* TO 'john'@'%' IDENTIFIED BY PASSWORD '*73A88CE13374128844E3ABB500F439A9C02C 53A3' *** 2. row *** Grants for john@%: GRANT SELECT ON `world`.* TO 'john'@'%' 96 Friday, 06 August 2010 96
  • 128. advanced command line techniques $_ getting query details 97 Friday, 06 August 2010 97
  • 129. which query is being executed? # wrong.sql # this script has one error use test ; drop table if exists t1 ; create table t1 (id int); insert into t1 values (a); 98 Friday, 06 August 2010 98
  • 130. which query is being executed? $ mysql < wrong.sql ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' # WHERE IS THE ERROR? 99 Friday, 06 August 2010 99
  • 131. which query is being executed? $ mysql -v < wrong.sql -------------- drop table if exists t1 -------------- -------------- create table t1 (id int) -------------- -------------- insert into t1 values (a) -------------- ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' 100 Friday, 06 August 2010 100
  • 132. getting execution times $ mysql -vv -e 'select rating,count(*) from sakila.film group by rating' +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 5 rows in set (0.00 sec) 101 Friday, 06 August 2010 101
  • 133. getting execution times (batch mode) $echo 'select rating,count(*) from sakila.film group by rating' | mysql -vv -------------- select rating,count(*) from sakila.film group by rating -------------- rating count(*) G 178 PG 194 PG-13 223 R 195 NC-17 210 5 rows in set 102 Friday, 06 August 2010 102
  • 134. getting execution times (batch mode) $ echo 'select rating,count(*) from sakila.film group by rating' | mysql -vvv -------------- select rating,count(*) from sakila.film group by rating -------------- +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 103 5 rows in set (0.00 sec) Friday, 06 August 2010 103
  • 135. advanced command line techniques getting _$ distinct results 104 Friday, 06 August 2010 104
  • 136. getting a single value $ mysql -e "show variables like 'port'" +---------------+-------+ | Variable_name | Value | +---------------+-------+ | port | 3306 | +---------------+-------+ # Here, we only want the port number 105 Friday, 06 August 2010 105
  • 137. getting a single value $ mysql -NB -e "show variables like 'port'" port 3306 # no header, batch mode # still two values 106 Friday, 06 August 2010 106
  • 138. getting a single value $ mysql -NB -e "show variables like 'port'" | awk '{print $2}' 3306 $ mysql -NB -e "show variables like 'port'" | perl -lane 'print $F[1]' 3306 # filtering the wanted value 107 Friday, 06 August 2010 107
  • 139. getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i slave_io_running Slave_IO_Running: Yes # too many things. # We only want a 'yes' or 'no' 108 Friday, 06 August 2010 108
  • 140. getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i "slave_io_running" | perl -lane 'print $F[1]' Yes # yes! 109 Friday, 06 August 2010 109
  • 141. What's perl -lane? 110 Friday, 06 August 2010 110
  • 142. What's perl -lane? perl -lane must turn perl 110 Friday, 06 August 2010 110
  • 143. Perl command line must-know facts (1) perl -e 'print "hellon"' # executes perl code 111 Friday, 06 August 2010 111
  • 144. Perl command line must-know facts (2) perl -ne 'print if /^a/' # starts a loop from standard input # and executes the perl code for each # iteration perl -MO=Deparse -ne 'print if /^a/' LINE: while (defined($_ = <ARGV>)) { print $_ if /^a/; } 112 Friday, 06 August 2010 112
  • 145. Perl command line must-know facts (3) perl -pe 's/(^aw+)/$1/' # like -ne, but prints each line $ perl -MO=Deparse -pe 's/(^aw+)/$1/' LINE: while (defined($_ = <ARGV>)) { s/(^aw+)/$1/; } continue { print $_; } 113 Friday, 06 August 2010 113
  • 146. Perl command line must-know facts (4) perl -le 'print "hello"' # strips newlines from input and adds # them to output $ perl -e 'print "hello"' hello$ $ perl -le 'print "hello"' hello $ 114 Friday, 06 August 2010 114
  • 147. Perl command line must-know facts (5) perl -lane 'print $F[0]' # splits input into @F array $ echo "a b c" | perl -lane 'print $F[1]' b $ echo "a b c" | perl -lane 'print $F[2]' c 115 Friday, 06 August 2010 115
  • 148. Perl command line must-know facts (6) Perl and shell contention for $ #!/bin/bash Take the $$$ and run 116 Friday, 06 August 2010 116
  • 149. Beware of $VARIABLES on the command line $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117
  • 150. double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117
  • 151. double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl can see it 117 Friday, 06 August 2010 117
  • 152. double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 117 Friday, 06 August 2010 117
  • 153. double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 single quotes the consumed by prevent variable shell, before Perl consumption by can shell it the see117 Friday, 06 August 2010 117
  • 154. advanced command line techniques combining column names 118 Friday, 06 August 2010 118
  • 155. combining column names $ mysql -N -e 'desc world.City' +-------------+----------+----+-----+ | ID | int(11) | NO | PRI | | Name | char(35) | NO | | | CountryCode | char(3) | NO | | | District | char(20) | NO | | | Population | int(11) | NO | | +-------------+----------+----+-----+ 119 Friday, 06 August 2010 119
  • 156. combining column names # Expected result: # SELECT CONCAT_WS('|', # ID,Name,CountryCode,District, # Population) from City 120 Friday, 06 August 2010 120
  • 157. combining column names (I) echo "select concat_ws('|', $(mysql -N -e 'desc world.City' | perl -lane 'push @f, $F[0];' -e 'END{print join ",", @f}')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 121 Friday, 06 August 2010 121
  • 158. combining column names (II) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="City" and table_schema="world"')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 122 Friday, 06 August 2010 122
  • 159. combining column names (IIa) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) from world.Country" select concat_ws('|', Code,Name,Continent,Region,SurfaceArea,Inde pYear,Population,LifeExpectancy,GNP,GNPOld, LocalName,GovernmentForm,HeadOfState,Capita l,Code2) from world.Country 123 Friday, 06 August 2010 123
  • 160. piping the result back to mysql echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) as line from world.Country" | mysql | head -n 4 line AFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919| 22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate| Mohammad Omar|1|AF NLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000| 78.3|371362.00|360478.00|Nederland|Constitutional Monarchy| Beatrix|5|NL ANT|Netherlands Antilles|North America|Caribbean|800.00|217000| 74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of 124 The Netherlands|Beatrix|33|AN Friday, 06 August 2010 124
  • 161. advanced command line techniques $_ dumping data 125 Friday, 06 August 2010 125
  • 162. advanced command line techniques $_ dumping data 125 Friday, 06 August 2010 125
  • 163. dumping a tab delimited data set # the naive way $ mysql -e "SELECT * into outfile '/tmp/city_data' from world.City" # it works only if: # - /tmp/city_data does not exist # - /tmp is world writable 126 Friday, 06 August 2010 126
  • 164. dumping a tab delimited data set $ mysql -B -N -e "SELECT * from world.City" > /tmp/city_data # it works always # if you omit the "-N" it also # includes a header 127 Friday, 06 August 2010 127
  • 165. dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", map {s/"/""/g; /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128
  • 166. dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", splits the line by tabs, instead map {s/"/""/g; of the default whitespace /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128
  • 167. advanced command line techniques script it 129 Friday, 06 August 2010 129
  • 168. use the mysql.sh script  The companion code for this tutorial  http://tinyurl.com/cli-tut-code-2010  It will save you much typing  But use it at your own risk! 130 Friday, 06 August 2010 130
  • 169. Usage • download the script • load it $ source mysql.sh 131 Friday, 06 August 2010 131
  • 170. mysql.sh functions • my_batch "query" • returns the result of the query. • Without header • Without box $ my_batch "SELECT * FROM test.t1" 1 aaa 2 bbb 3 ccc 132 Friday, 06 August 2010 132
  • 171. mysql.sh functions • my_value "what" • returns a single value • e.g. $ my_value "version()" 5.1.49 $ my_value "@@server_id" 1 133 Friday, 06 August 2010 133
  • 172. mysql.sh functions • my_var "what" • returns the value of a given variable • e.g. $ my_var port 3306 $ my_var datadir /usr/local/mysql/data 134 Friday, 06 August 2010 134
  • 173. mysql.sh functions • my_status "what" • returns the value of a given status item • e.g. $ my_status com_select 2653 $ my_status connections 452 135 Friday, 06 August 2010 135
  • 174. mysql.sh functions • my_dump_dataset "query" [header] • returns the values of a given query separated by tabs • if a header is requested, it will be part of the result • e.g. $ my_dump_dataset "select * from world.City" | head -n 5 1 Kabul AFG Kabol 1780000 2 Qandahar AFG Qandahar 237500 3 Herat AFG Herat 186800 4 Mazar-e-Sharif AFG Balkh 127800 136 5 Amsterdam NLD Noord-Holland 731200 Friday, 06 August 2010 136
  • 175. mysql.sh functions • my_csv "query" [separator] • returns the values of a given query separated by commas, or by a given separator • e.g. $ my_csv 'select * from world.City' | head -n 3 "ID","Name","CountryCode","District"," Population" 1,"Kabul","AFG","Kabol",1780000 2,"Qandahar","AFG","Qandahar",237500 137 Friday, 06 August 2010 137
  • 176. inside the command line client handling :-) @ @ $_ output 138 Friday, 06 August 2010 138
  • 177. pager mysql> SELECT * FROM world.City; rec … rec … rec … 4000 + records unreadable 139 Friday, 06 August 2010 139
  • 178. pager mysql> pager less PAGER set to 'less' mysql> SELECT * FROM world.City; # [ records in a manageable window ] mysql> nopager PAGER set to stdout # back to normal 140 Friday, 06 August 2010 140
  • 179. More pager tricks mysql> pager vim - # like 'less', but you can also edit # the results and eventually save # them 141 Friday, 06 August 2010 141
  • 180. More pager tricks mysql> pager md5sum PAGER set to 'md5sum' mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - 4079 rows in set (0.01 sec) # produce a CRC for the result set # instead of displaying it. # Useful for data comparison. # (Thanks to Baron Schwartz 142 Friday, 06 August 2010 142
  • 181. Comparing data sets with md5sum mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by ID; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by name; 5ab980f5b69a6e65d3b8cd5a95ec94e0 - mysql> select * from City order by District; 2bf37cbe91b3ffeaebc8c4b6084b8784 - 143 Friday, 06 August 2010 143
  • 182. Comparing data sets with md5sum mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by ID; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by name; 5ab980f5b69a6e65d3b8cd5a95ec94e0 - mysql> select * from City order by District; 2bf37cbe91b3ffeaebc8c4b6084b8784 - 143 Friday, 06 August 2010 143
  • 183. More pager tricks mysql> pager cat > /tmp/t1.txt mysql> select "one" union select "two"; mysql> pager cat > /tmp/t2.txt mysql> select "one" union select "TWO"; mysql> nopager mysql> ! vimdiff -o /tmp/t[12].txt # see the difference between two # datasets 144 Friday, 06 August 2010 144
  • 185. logging sessions mysql> tee mylog.txt # logs everything that you see on # the screen. # your commands, the results, # warnings, error messages, and so on. mysql> notee # back to normal 146 Friday, 06 August 2010 146
  • 186. Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 147
  • 187. read more  http://datacharmer.blogspot.com Friday, 06 August 2010 148
  • 188. THANKS! ? Question time Comments on Twitter: @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 149