This document provides an overview of the command line and related tools for database administrators (DBAs). It discusses basics like the shell and command syntax. It introduces common command line tools like MySQL, Perl, awk and sort. It explains Unix concepts like pipes for parallel processing. It also discusses who receives output from commands - the shell, applications or database servers. Finally, it briefly mentions the readline feature for command line interactions.
Shooting from the hip - Perl at the command line for DBAs
1. Shooting from the hip.
Perl at the command
:)
@ @
line for DBAs
-
$_
#!
Giuseppe Maxia
MySQL Community Team Lead
@datacharmer
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Friday, 06 August 2010 1
2. about me -Giuseppe Maxia
a.k.a. The Data Charmer
MySQL Community Team Lead
Long time hacking with MySQL features
Formerly, database consultant, designer, coder.
A passion for QA
An even greater passion for open source
... and community
Passionate blogger
http://datacharmer.blogspot.com
Friday, 06 August 2010 2
3. Updated presentation slides
you will find an up to date copy of these slides at:
http://tinyurl.com/shoot-cl-2010
http://tinyurl.com/shoot-cl-full-2010
Friday, 06 August 2010 3
4. I love your feedback!
Tell it to the Twitter community
#yapc_eu "your feedback here"
Twitter me
@datacharmer "your feedback here"
Blog about it
Friday, 06 August 2010 4
5. This is a story about
languages
Friday, 06 August 2010 5
6. you got to understand,
Bernie, it's not personal.
It's just that you are
obsolete
We use
language
to talk to
computers
Friday, 06 August 2010 6
7. 10 print "hello"
We use 20 goto 10
languages
to talk to
computers
Friday, 06 August 2010 7
15. I speak
Perl
with a hint of a C
accent
Friday, 06 August 2010 12
16. I speak
French
and
Spanish
Friday, 06 August 2010 13
17. I speak
French
and
Spanish
but since I use
them very little,
they have melted
with Italian and
English
Friday, 06 August 2010 13
18. I speak
#!/bin/bash
Bash
Friday, 06 August 2010 14
19. I speak
#!/bin/bash
Bash
with a strong
Perl accent
Friday, 06 August 2010 14
20. I speak fluent Italian
La donzelletta vien dalla
campagna, in sul calar del
sole, col suo fascio dell'erba
e reca in mano un mazzolino
di rose e viole, onde, siccome
suole, ornare ella si appresta ...
Friday, 06 August 2010 15
21. However, Sardinian comes handy sometimes
Maissavida
ti frimisti! (*)
(*) May you never stop
Friday, 06 August 2010 16
22. I speak fluent English
When in the Course of human events, it becomes
necessary for one people to dissolve the political bands
which have connected them with another, and to assume
among the powers of the earth, the separate and equal
station to which the Laws of Nature and of Nature's God
entitle them, a decent respect to the opinions of mankind
requires that they should declare the causes which impel
them to the separation.
Friday, 06 August 2010 17
23. I speak fluent English
When in the Course of human events, it becomes
necessary for one people to dissolve the political bands
which have connected them with another, and to assume
among the powers of the earth, the separate and equal
station to which the Laws of Nature and of Nature's God
entitle them, a decent respect to the opinions of mankind
requires that they should declare the causes which impel
them to the separation.
hmm.
wrong
flag
Friday, 06 August 2010 17
24. I speak fluent English
When in the Course of human events, it becomes
necessary for one people to dissolve the political bands
which have connected them with another, and to assume
among the powers of the earth, the separate and equal
station to which the Laws of Nature and of Nature's God
entitle them, a decent respect to the opinions of mankind
requires that they should declare the causes which impel
them to the separation.
Friday, 06 August 2010 17
25. But sometimes Italian is still necessary
barista al fresco pasta
maestro espresso tempo
vendetta soprano finale piano
dilettante cappuccino stucco
stiletto virtuoso fiasco presto lotto
pizza bravo lasagna
studio spaghetti
stanza
Friday, 06 August 2010 18
26. And sometimes Italian is needed
I tuoi perversi
defunti!(*)
(*) Your wicked ancestors!
Friday, 06 August 2010 19
27. Although Sardinian is still effective here
Ancu ti currullidi
sa giustizia (*)
(*) May the rule of law chase you
Friday, 06 August 2010 20
28. I speak Perl fluently
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
while (my $line = <stdin>){
# …
Friday, 06 August 2010 21
29. Sometimes, I need help from C
use Inline C;
print "9 + 16 = ", add(9, 16), "n";
__END__
__C__
int add(int x, int y) {
return x + y;
}
Friday, 06 August 2010 22
30. Most of my work at the command line is Bash
$ for D in $HOME/*/; do du -sh $D;
> done > /tmp/occupancy
#!/bin/bash
Friday, 06 August 2010 23
31. And sometimes, a pinch of Perl can help
$ perl -pe
's{(dd)-(dd)-(dddd)}{$3-$1-$2}g'
< rawfile.txt > betterfile.txt
#!/bin/bash
Friday, 06 August 2010 24
32. And then on top of all this, I need to speak SQL
$ mysql -e
'SELECT COUNT(*) FROM dbname.tbl_name
WHERE col_name = 1 and other_col="A" '
Friday, 06 August 2010 25
33. I wish I could say that my SQL has a Perlish
accent
$ mysql -e
'SELECT COUNT(*) FROM dbname.tbl_name
WHERE col_name = 1 and other_col="A" '
use strict;
use DBI;?
Friday, 06 August 2010 26
34. I wish I could say that my SQL has a Perlish
accent
$ mysql -e
'SELECT COUNT(*) FROM dbname.tbl_name
WHERE col_name = 1 and other_col="A" '
but it wouldn't be true
Friday, 06 August 2010 27
35. I could curse in Perl
$ perl -pe
's{(dd)-(dd)-(ddd
d)}{$3-$1-$2}g'
< rawfile.txt >
betterfile.txt (*)
But it would not
get me far.
(*) already translated
Friday, 06 August 2010 28
37. Then
I will need to mix languages.
Friday, 06 August 2010 29
38. Then
I will need to mix languages.
This talk will show exactly that.
Friday, 06 August 2010 29
39. Then
I will need to mix languages.
This talk will show exactly that.
Mixing languages.
Friday, 06 August 2010 29
40. Then
I will need to mix languages.
This talk will show exactly that.
Mixing languages.
Yes.
Friday, 06 August 2010 29
41. Then
I will need to mix languages.
This talk will show exactly that.
Mixing languages.
Yes.
$ at the command line
Friday, 06 August 2010 29
42. What this tutorial is about
Providing tools and tips for command line users;
Learn a few tricks to make you achieve more and type
less.
Friday, 06 August 2010 30
43. What this tutorial is about
Providing tools and tips for command line users;
Learn a few tricks to make you achieve more and type
less.
Making GUI enthusiasts green with envy;
bar
foo
baz
Friday, 06 August 2010 30
44. the command line (for rookies)
BASICS
31
Friday, 06 August 2010 31
45. The shell
• It's a command interpreter
• There are many flavors
> bash
> zsh
> ksh
32
Friday, 06 August 2010 32
46. MySQL command line client
• called "mysql"
> Takes SQL commands
> sends them to the server
> gets results
• but it can do much more ...
33
Friday, 06 August 2010 33
47. other tools
• Shell scripts
• Aliases
• Perl
• awk
• sort
• …. and a lot of Unix little helper
34
Friday, 06 August 2010 34
48. other tools
• Shell scripts
• Aliases
• Perl which could replace everything else, of course
• awk
• sort
• …. and a lot of Unix little helper
34
Friday, 06 August 2010 34
49. in case it wasn't clear (yeah, for
rookies)
35
Friday, 06 August 2010 35
50. in case it wasn't clear (yeah, for
rookies)
• command line actions are
35
Friday, 06 August 2010 35
51. in case it wasn't clear (yeah, for
rookies)
• command line actions are
> commands (the name of the application to invoke)
35
Friday, 06 August 2010 35
52. in case it wasn't clear (yeah, for
rookies)
• command line actions are
> commands (the name of the application to invoke)
> parameters (optional arguments for the application)
35
Friday, 06 August 2010 35
53. in case it wasn't clear (yeah, for
rookies)
• command line actions are
> commands (the name of the application to invoke)
> parameters (optional arguments for the application)
> and the ENTER key
35
Friday, 06 August 2010 35
54. in case it wasn't clear (yeah, for
rookies)
• command line actions are
> commands (the name of the application to invoke)
> parameters (optional arguments for the application)
> and the ENTER key
Enter
35
Friday, 06 August 2010 35
55. The Unix philosophy (1)
• One tool for every feature
> make one tool that does one thing well
> for many features, make many tools
> make them work together (see next slide)
36
Friday, 06 August 2010 36
56. The Unix philosophy (2)
• To make several tool cooperate:
> pipe the output of one tool to the input of another one.
> for more actions, pipe more tools
$ sort < filename | uniq | nl
37
Friday, 06 August 2010 37
71. Who gets what
• shell
• application
• database server
• pager
40
Friday, 06 August 2010 40
72. who gets what
shell prompt
$ mysql
mysql client
mysql server
41
Friday, 06 August 2010 41
73. who gets what
any host mysql server
shell
mysql client
42
Friday, 06 August 2010 42
74. who gets what
$ mysql -u username -ppassword
First the application
then the server
43
Friday, 06 August 2010 43
75. who gets what
$ mysql -u username -ppassword
-e "select VERSION()"
goes to the client, then the
server
goes to the server
44
Friday, 06 August 2010 44
76. who gets what
mysql> help
mysql> pager less goes to the client only
mysql> select 1;
goes to the server
45
Friday, 06 August 2010 45
78. readline
• Common component
> shell
> mysql client
> many CLI applications
47
Friday, 06 August 2010 47
79. Main cursor movements
Ctrl A Start/End of line
Ctrl E
Meta B Back/Forward one word
Meta F
previous/next command
Ctrl L clear screen and take
current line on top
48
Friday, 06 August 2010 48
80. Changing text
Meta U Uppercase word
Meta L Lowercase word
Ctrl D delete word
Ctrl K delete to end of line
Ctrl - undo
for more tips:
$ man readline 49
Friday, 06 August 2010 49
81. Some conventions in this talk
$ command
at the shell
mysql> command
at the mysql prompt
50
Friday, 06 August 2010 50
88. nesting variables
# t.sh
CMD="SELECT TABLE_NAME"
TBL='information_schema.tables'
WHC1="TABLE_SCHEMA='mysql'"
WHC2="TABLE_NAME like 'u%' "
Q1="$CMD FROM $TBL"
Q2="WHERE $WHC1 AND $WHC2"
Q="$Q1 $Q2"
echo $Q
echo $Q | mysql
57
Friday, 06 August 2010 57
89. nesting variables
$ sh t.sh
SELECT TABLE_NAME FROM
information_schema.tables WHERE
TABLE_SCHEMA='mysql' AND TABLE_NAME like
'u%'
TABLE_NAME
user
58
Friday, 06 August 2010 58
91. SQL commands from a file
$ mysql < filename.sql
mysql> source filename.sql
• executes the commands contained in
filename.sql
• commands can be queries understood
by the server
• and commands for the client
60
Friday, 06 August 2010 60
92. SQL commands at the command line
$ application | mysql
$ echo "SELECT 1" | mysql
$ mysql -e "SELECT 1"
• executes the commands from standard
input
• commands can be queries understood
by the server
• and commands for the client
61
Friday, 06 August 2010 61
93. batch mode
$ echo "select curdate()" | mysql
curdate()
2010-04-12
62
Friday, 06 August 2010 62
96. Forcing batch mode
mysql -B -e 'select curdate()'
curdate()
2010-04-12
65
Friday, 06 August 2010 65
97. Forcing batch mode - no header
mysql -N -B -e 'select curdate()'
2010-04-12
ANT!
PO RT
IM
66
Friday, 06 August 2010 66
98. Shell + mysql
$ echo "The current server's date is
`mysql -N -B -e 'select curdate()'`."
$ echo "The current server's date is
$(mysql -N -B -e 'select curdate
()')."
67
Friday, 06 August 2010 67
99. advanced command line techniques
((((((
(:
@ @
-
$_
loops
68
Friday, 06 August 2010 68
100. loops basics
$ for N in 1 2 3
> do
> echo $N
> done
1
2
3
69
Friday, 06 August 2010 69
101. loops basics
$ for N in 1 2 3 ; do echo $N ; done
1
2
3
70
Friday, 06 August 2010 70
102. loops basics
$ for N in $(seq 1 3) ;
do echo "this is number $N" ;
done
this is number 1
this is number 2
this is number 3
71
Friday, 06 August 2010 71
103. loops from file
$ head /usr/share/dict/words
A
a
aa
aal
aalii
aam
Aani
aardvark
aardwolf
72
Friday, 06 August 2010 72
104. loops from file
$ for U in
$(head /usr/share/dict/words ) ; do
perl -le 'print uc shift' $U ; done
A
A
AA
AAL
AALII
AAM
AANI
73
Friday, 06 August 2010 73
105. loops from mysql results
$ mysql -B -N -e 'show tables from
world'
City
Country
CountryLanguage
74
Friday, 06 August 2010 74
106. loops from mysql results
$ for T in $(mysql -B -N -e 'show
tables from world') ;
> do echo "CREATE TABLE world2.$T
LIKE world.$T;" ; done
CREATE TABLE world2.City LIKE
world.City;
CREATE TABLE world2.Country LIKE
world.Country;
CREATE TABLE world2.CountryLanguage
LIKE world.CountryLanguage;
75
Friday, 06 August 2010 75
107. piping loops back into mysql
$ for T in $(mysql -B -N -e 'show
tables from world') ;
> do echo "CREATE TABLE world2.$T
LIKE world.$T;" ; done
> | mysql -vv
76
Friday, 06 August 2010 76
109. doing something for every table
in every database
$ mysql -e "show databases"
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| sakila |
| world |
+--------------------+
78
Friday, 06 August 2010 78
110. doing something for every table
in every database
$ mysql -e "show tables from world"
+-----------------+
| Tables_in_world |
+-----------------+
| City |
| Country |
| CountryLanguage |
+-----------------+
79
Friday, 06 August 2010 79
112. actual code
alias mybatch='mysql -B -N'
for D in $(mybatch -e "SHOW SCHEMAS")
do
for T in
$(mybatch -e "SHOW TABLES FROM $D")
do
echo "$D.$T"
done
done
81
Friday, 06 August 2010 81
115. Custom option files
$ mysql --no-defaults
$ mysql --defaults-file=filename
$ mysql --defaults-extra-file=filename
84
Friday, 06 August 2010 84
116. Option files sections
[mysqld]
# option for server
[client]
# option for any mysql client
[mysql]
# options specific to 'mysql' CLI app
85
Friday, 06 August 2010 85
123. Another useful alias
alias mybatch='mysql -B -N '
mybatch -e "select version()"
5.1.45-log
92
Friday, 06 August 2010 92
124. ( )
advanced command line techniques
((((((
(:
@ @
-
#!
functions
93
Friday, 06 August 2010 93
125. functions
• like aliases, but with parameters
94
Friday, 06 August 2010 94
126. sample function
function grant_new {
mysql_as_root -e
"GRANT $1 on $2 to $3 identified
by "$4" "
}
$ grant_new SELECT "world.*" john
john_secret
95
Friday, 06 August 2010 95
127. sample function
$ mysql_as_root
-e 'show grants for johnG'
*** 1. row ***
Grants for john@%: GRANT USAGE ON *.*
TO 'john'@'%' IDENTIFIED BY PASSWORD
'*73A88CE13374128844E3ABB500F439A9C02C
53A3'
*** 2. row ***
Grants for john@%: GRANT SELECT ON
`world`.* TO 'john'@'%'
96
Friday, 06 August 2010 96
128. advanced command line techniques
$_ getting
query
details
97
Friday, 06 August 2010 97
129. which query is being executed?
# wrong.sql
# this script has one error
use test ;
drop table if exists t1 ;
create table t1 (id int);
insert into t1 values (a);
98
Friday, 06 August 2010 98
130. which query is being executed?
$ mysql < wrong.sql
ERROR 1054 (42S22) at line 4: Unknown
column 'a' in 'field list'
# WHERE IS THE ERROR?
99
Friday, 06 August 2010 99
131. which query is being executed?
$ mysql -v < wrong.sql
--------------
drop table if exists t1
--------------
--------------
create table t1 (id int)
--------------
--------------
insert into t1 values (a)
--------------
ERROR 1054 (42S22) at line 4: Unknown column
'a' in 'field list'
100
Friday, 06 August 2010 100
132. getting execution times
$ mysql -vv -e
'select rating,count(*) from sakila.film
group by rating'
+--------+----------+
| rating | count(*) |
+--------+----------+
| G | 178 |
| PG | 194 |
| PG-13 | 223 |
| R | 195 |
| NC-17 | 210 |
+--------+----------+
5 rows in set (0.00 sec) 101
Friday, 06 August 2010 101
133. getting execution times (batch mode)
$echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vv
--------------
select rating,count(*) from sakila.film group by
rating
--------------
rating count(*)
G 178
PG 194
PG-13 223
R 195
NC-17 210
5 rows in set
102
Friday, 06 August 2010 102
134. getting execution times (batch mode)
$ echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vvv
--------------
select rating,count(*) from sakila.film group by
rating
--------------
+--------+----------+
| rating | count(*) |
+--------+----------+
| G | 178 |
| PG | 194 |
| PG-13 | 223 |
| R | 195 |
| NC-17 | 210 |
+--------+----------+
103
5 rows in set (0.00 sec)
Friday, 06 August 2010 103
135. advanced command line techniques
getting
_$
distinct
results
104
Friday, 06 August 2010 104
136. getting a single value
$ mysql -e "show variables like 'port'"
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| port | 3306 |
+---------------+-------+
# Here, we only want the port number
105
Friday, 06 August 2010 105
137. getting a single value
$ mysql -NB -e "show variables like 'port'"
port 3306
# no header, batch mode
# still two values
106
Friday, 06 August 2010 106
138. getting a single value
$ mysql -NB
-e "show variables like 'port'"
| awk '{print $2}'
3306
$ mysql -NB
-e "show variables like 'port'"
| perl -lane 'print $F[1]'
3306
# filtering the wanted value
107
Friday, 06 August 2010 107
139. getting a single value: is replication
working?
$ mysql
-e "show slave status G"
| grep -i slave_io_running
Slave_IO_Running: Yes
# too many things.
# We only want a 'yes' or 'no'
108
Friday, 06 August 2010 108
140. getting a single value: is replication
working?
$ mysql
-e "show slave status G"
| grep -i "slave_io_running"
| perl -lane 'print $F[1]'
Yes
# yes!
109
Friday, 06 August 2010 109
144. Perl command line must-know facts (2)
perl -ne 'print if /^a/'
# starts a loop from standard input
# and executes the perl code for each
# iteration
perl -MO=Deparse -ne 'print if /^a/'
LINE: while (defined($_ = <ARGV>)) {
print $_ if /^a/;
}
112
Friday, 06 August 2010 112
145. Perl command line must-know facts (3)
perl -pe 's/(^aw+)/$1/'
# like -ne, but prints each line
$ perl -MO=Deparse -pe 's/(^aw+)/$1/'
LINE: while (defined($_ = <ARGV>)) {
s/(^aw+)/$1/;
}
continue {
print $_;
}
113
Friday, 06 August 2010 113
146. Perl command line must-know facts (4)
perl -le 'print "hello"'
# strips newlines from input and adds
# them to output
$ perl -e 'print "hello"'
hello$
$ perl -le 'print "hello"'
hello
$
114
Friday, 06 August 2010 114
147. Perl command line must-know facts (5)
perl -lane 'print $F[0]'
# splits input into @F array
$ echo "a b c" |
perl -lane 'print $F[1]'
b
$ echo "a b c" |
perl -lane 'print $F[2]'
c
115
Friday, 06 August 2010 115
148. Perl command line must-know facts (6)
Perl and shell contention for $
#!/bin/bash
Take the $$$ and run
116
Friday, 06 August 2010 116
149. Beware of $VARIABLES on the
command line
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
117
Friday, 06 August 2010 117
150. double quotes
Beware of $VARIABLES on theconsumed by the
allow the shell to
shell, before Perl
interpolate
command line can see it
variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
117
Friday, 06 August 2010 117
151. double quotes
Beware of $VARIABLES on theconsumed by the
allow the shell to
shell, before Perl
interpolate
command line can see it
variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
consumed by the
shell, before Perl
can see it
117
Friday, 06 August 2010 117
152. double quotes
Beware of $VARIABLES on the consumed by the
allow the shell to
shell, before Perl
interpolate
command line can see it
variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
consumed by the
shell, before Perl
$ perl -le '$x=1; print $x' can see it
1
117
Friday, 06 August 2010 117
153. double quotes
Beware of $VARIABLES on the consumed by the
allow the shell to
shell, before Perl
interpolate
command line can see it
variables
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.
consumed by the
shell, before Perl
$ perl -le '$x=1; print $x' can see it
1
single quotes the
consumed by
prevent variable
shell, before Perl
consumption by
can shell it
the see117
Friday, 06 August 2010 117
154. advanced command line techniques
combining
column
names
118
Friday, 06 August 2010 118
155. combining column names
$ mysql -N -e 'desc world.City'
+-------------+----------+----+-----+
| ID | int(11) | NO | PRI |
| Name | char(35) | NO | |
| CountryCode | char(3) | NO | |
| District | char(20) | NO | |
| Population | int(11) | NO | |
+-------------+----------+----+-----+
119
Friday, 06 August 2010 119
156. combining column names
# Expected result:
# SELECT CONCAT_WS('|',
# ID,Name,CountryCode,District,
# Population) from City
120
Friday, 06 August 2010 120
158. combining column names (II)
echo "select concat_ws('|', $(
mysql -NB -e 'select group_concat
(column_name) from
information_schema.columns where
table_name="City" and
table_schema="world"')) from
world.City"
select concat_ws('|',
ID,Name,CountryCode,District,Populatio
n) from world.City
122
Friday, 06 August 2010 122
159. combining column names (IIa)
echo "select concat_ws('|', $(
mysql -NB -e 'select group_concat
(column_name) from
information_schema.columns where
table_name="Country" and
table_schema="world"')) from
world.Country"
select concat_ws('|',
Code,Name,Continent,Region,SurfaceArea,Inde
pYear,Population,LifeExpectancy,GNP,GNPOld,
LocalName,GovernmentForm,HeadOfState,Capita
l,Code2) from world.Country 123
Friday, 06 August 2010 123
160. piping the result back to mysql
echo "select concat_ws('|', $(
mysql -NB -e 'select group_concat
(column_name) from
information_schema.columns where
table_name="Country" and
table_schema="world"')) as line from
world.Country" | mysql | head -n 4
line
AFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919|
22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate|
Mohammad Omar|1|AF
NLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000|
78.3|371362.00|360478.00|Nederland|Constitutional Monarchy|
Beatrix|5|NL
ANT|Netherlands Antilles|North America|Caribbean|800.00|217000|
74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of
124
The Netherlands|Beatrix|33|AN
Friday, 06 August 2010 124
163. dumping a tab delimited data set
# the naive way
$ mysql
-e "SELECT *
into outfile '/tmp/city_data'
from world.City"
# it works only if:
# - /tmp/city_data does not exist
# - /tmp is world writable
126
Friday, 06 August 2010 126
164. dumping a tab delimited data set
$ mysql -B -N
-e "SELECT * from world.City"
> /tmp/city_data
# it works always
# if you omit the "-N" it also
# includes a header
127
Friday, 06 August 2010 127
165. dumping a CSV
$ mysql -B
-e "SELECT * from world.City"
| perl -F"t" -lane
'print join ",",
map {s/"/""/g;
/^[d.]+$/ ? $_ : qq("$_")}
@F '
> /tmp/city_data
128
Friday, 06 August 2010 128
166. dumping a CSV
$ mysql -B
-e "SELECT * from world.City"
| perl -F"t" -lane
'print join ",", splits the line by tabs, instead
map {s/"/""/g; of the default whitespace
/^[d.]+$/ ? $_ : qq("$_")}
@F '
> /tmp/city_data
128
Friday, 06 August 2010 128
168. use the mysql.sh script
The companion code for this tutorial
http://tinyurl.com/cli-tut-code-2010
It will save you much typing
But use it at your own risk!
130
Friday, 06 August 2010 130
169. Usage
• download the script
• load it
$ source mysql.sh
131
Friday, 06 August 2010 131
170. mysql.sh functions
• my_batch "query"
• returns the result of the query.
• Without header
• Without box
$ my_batch "SELECT * FROM test.t1"
1 aaa
2 bbb
3 ccc
132
Friday, 06 August 2010 132
171. mysql.sh functions
• my_value "what"
• returns a single value
• e.g.
$ my_value "version()"
5.1.49
$ my_value "@@server_id"
1
133
Friday, 06 August 2010 133
172. mysql.sh functions
• my_var "what"
• returns the value of a given variable
• e.g.
$ my_var port
3306
$ my_var datadir
/usr/local/mysql/data
134
Friday, 06 August 2010 134
173. mysql.sh functions
• my_status "what"
• returns the value of a given status item
• e.g.
$ my_status com_select
2653
$ my_status connections
452
135
Friday, 06 August 2010 135
174. mysql.sh functions
• my_dump_dataset "query" [header]
• returns the values of a given query separated by tabs
• if a header is requested, it will be part of the result
• e.g.
$ my_dump_dataset "select * from
world.City" | head -n 5
1 Kabul AFG Kabol 1780000
2 Qandahar AFG Qandahar 237500
3 Herat AFG Herat 186800
4 Mazar-e-Sharif AFG Balkh 127800
136
5 Amsterdam NLD Noord-Holland 731200
Friday, 06 August 2010 136
175. mysql.sh functions
• my_csv "query" [separator]
• returns the values of a given query separated by
commas, or by a given separator
• e.g.
$ my_csv 'select * from world.City' |
head -n 3
"ID","Name","CountryCode","District","
Population"
1,"Kabul","AFG","Kabol",1780000
2,"Qandahar","AFG","Qandahar",237500
137
Friday, 06 August 2010 137
176. inside the command line client
handling
:-)
@ @
$_
output
138
Friday, 06 August 2010 138
177. pager
mysql> SELECT * FROM world.City;
rec …
rec …
rec …
4000 + records
unreadable
139
Friday, 06 August 2010 139
178. pager
mysql> pager less
PAGER set to 'less'
mysql> SELECT * FROM world.City;
# [ records in a manageable window ]
mysql> nopager
PAGER set to stdout
# back to normal
140
Friday, 06 August 2010 140
179. More pager tricks
mysql> pager vim -
# like 'less', but you can also edit
# the results and eventually save
# them
141
Friday, 06 August 2010 141
180. More pager tricks
mysql> pager md5sum
PAGER set to 'md5sum'
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543 -
4079 rows in set (0.01 sec)
# produce a CRC for the result set
# instead of displaying it.
# Useful for data comparison.
# (Thanks to Baron Schwartz
142
Friday, 06 August 2010 142
181. Comparing data sets with md5sum
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by ID;
5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by name;
5ab980f5b69a6e65d3b8cd5a95ec94e0 -
mysql> select * from City order by
District;
2bf37cbe91b3ffeaebc8c4b6084b8784 - 143
Friday, 06 August 2010 143
182. Comparing data sets with md5sum
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by ID;
5d17ffa50d6da200dee621823ade2543 -
mysql> select * from City order by name;
5ab980f5b69a6e65d3b8cd5a95ec94e0 -
mysql> select * from City order by
District;
2bf37cbe91b3ffeaebc8c4b6084b8784 - 143
Friday, 06 August 2010 143
183. More pager tricks
mysql> pager cat > /tmp/t1.txt
mysql> select "one" union select
"two";
mysql> pager cat > /tmp/t2.txt
mysql> select "one" union select
"TWO";
mysql> nopager
mysql> ! vimdiff -o /tmp/t[12].txt
# see the difference between two
# datasets
144
Friday, 06 August 2010 144
185. logging sessions
mysql> tee mylog.txt
# logs everything that you see on
# the screen.
# your commands, the results,
# warnings, error messages, and so on.
mysql> notee
# back to normal
146
Friday, 06 August 2010 146
186. Updated presentation slides
you will find an up to date copy of these slides at:
http://tinyurl.com/shoot-cl-2010
http://tinyurl.com/shoot-cl-full-2010
Friday, 06 August 2010 147
187. read more
http://datacharmer.blogspot.com
Friday, 06 August 2010 148
188. THANKS!
?
Question
time
Comments on Twitter:
@datacharmer
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Friday, 06 August 2010 149