Solving Common Sql Problems With The Seq Engine

Copyright © 2009 Beat Vontobel
This work is made available under the Creative Commons Attribution-Noncommercial-Share Alike license, see
http://creativecommons.org/licenses/by-nc-sa/3.0/

Solving Common
SQL Problems
with the
SeqEngine
Beat Vontobel, CTO, MeteoNews AG
b.vontobel@meteonews.ch

http://seqengine.org

Solving SQL Problems with the SeqEngine
How to benefit from simple auxiliary tables holding
•
sequences
Use of a pluggable storage engine to create such tables
•

On the side:
•

Some interesting benchmarks
‣

MySQL-Optimizer caveats
‣

Remember once more how to do things the „SQL-way“
‣

Beat Vontobel
CTO, MeteoNews AG

Sequences: What are we talking about?
CREATE TABLE integer_sequence (
i INT NOT NULL PRIMARY KEY
);

INSERT INTO integer_sequence (i)
VALUES (1), (2), (3), (4), (5), (6), (7), (8);

SELECT * FROM integer_sequence;
+---+
|i|
+---+
|1|
|2|
|3|
|4|
…

Beat Vontobel
CTO, MeteoNews AG

Names used by others…
Pivot Table
•

Can be used to „pivot“ other tables („turn them around“)
‣

Integers Table
•

They often hold integers as data type
‣

Auxiliary/Utility Table
•

They help us solve problems, but contain no actual data
‣

Sequence Table
•

Just what it is: The name I‘ll use
‣

Beat Vontobel
CTO, MeteoNews AG

What we‘re not talking about (1)
-- Oracle Style Sequences
--
-- (mostly used to generate primary keys, much
-- like what MySQL‘s auto_increment feature is
-- used for)

CREATE SEQUENCE customers_seq
START WITH 1000
INCREMENT BY 1;

INSERT INTO customers (customer_id, name)
VALUES (customers_seq.NEXTVAL, 'John Doe');

Beat Vontobel
CTO, MeteoNews AG

What we‘re not talking about (2)
Sequence in Mathematics:
•

„an ordered list of objects“
‣

n-tuple
‣

Sequence Table in SQL:
•

a set, unordered by definition
‣
F = {n | 1 ≤ n ≤ 20; n is integer}
relation (set of 1-tuples)
‣

Beat Vontobel
CTO, MeteoNews AG

„Using such a utility table is a favorite old trick of experienced
SQL developers“

(Stéphane Faroult: The Art of SQL)

Beat Vontobel
CTO, MeteoNews AG

Finding Holes… typically Swiss!

Beat Vontobel
CTO, MeteoNews AG

Finding Holes… in a Table!
+---------+---------------------+------+---
| stat_id | datetime | tt |…
+---------+---------------------+------+------+
|… … | … …
| |
| ABO | 2004-11-03 22:40:00 | 8.3 | …
| ABO | 2004-11-03 22:50:00 | 8.7 |
| ABO | 2004-11-03 23:00:00 | 9.9 |
| ABO | 2004-11-03 23:10:00 | 7.8 |
| ABO | 2004-11-04 00:10:00 | 9.2 |
| ABO | 2004-11-04 00:20:00 | 9.1 |
| ABO | 2004-11-04 00:30:00 | 10.2 |
| ABO | 2004-11-04 00:40:00 | 9.3 |
| |
… |… … …
| | | |
+---------+---------------------+------+----
Beat Vontobel
CTO, MeteoNews AG

The table‘s create statement used for demo
CREATE TABLE temperatures (

stat_id CHAR(3) NOT NULL,
datetime TIMESTAMP NOT NULL,
tt decimal(3,1) DEFAULT NULL,

PRIMARY KEY (stat_id, datetime),
UNIQUE KEY reverse_primary (datetime, stat_id)

);

Beat Vontobel
CTO, MeteoNews AG

How to „SELECT“ a row that doesn‘t exist?
SELECT only returns rows that are there
•

WHERE only filters rows
•

We need something to generate rows!
•

Beat Vontobel
CTO, MeteoNews AG

Finding Holes… the naïve way
for(„all timestamps to check“) {

/* Single SELECTs for every timestamp */
db_query(„SELECT COUNT(*)
FROM temperatures
WHERE stat_id = ? AND datetime = ?“);

if(„no row found“) {
warn_about_missing_row(„timestamp“);
}
}

Beat Vontobel
CTO, MeteoNews AG

Finding Holes… the „standard“ way
/* Working with an ordered set */
db_query(„SELECT datetime
FROM temperatures
WHERE stat_id = ?
ORDER BY datetime ASC“);

for(„all timestamps to check“) {

db_fetch_row();

while(„timestamps don‘t match“) {
warn_about_missing_row();
increment_timestamp();
}
}

Beat Vontobel
CTO, MeteoNews AG

These were just ordinary JOINs!
for-Loop just walks an „imaginary“ timestamps table with a
•
sequence of all the values to check for!
Then we LEFT JOIN these timestamps against our
•
temperatures
or do a NOT EXIST subquery
‣

So, if we had a sequence table…
•

Beat Vontobel
CTO, MeteoNews AG

Sequence of timestamps
CREATE TABLE timestamps (
datetime TIMESTAMP NOT NULL PRIMARY KEY
);

INSERT INTO timestamps (datetime)
VALUES ('2004-01-01 00:00:00'),
('2004-01-01 00:00:10'),
…;

SELECT * FROM timestamps;
+---------------------+
| datetime |
+---------------------+
| 2004-01-01 00:00:00 |
| 2004-01-01 00:00:10 |
|… |

Beat Vontobel
CTO, MeteoNews AG

Queries using the sequence
SELECT *
FROM timestamps -- our „for-Loop“
LEFT JOIN temperatures
ON timestamps.datetime = temperatures.datetime
WHERE temperatures.datetime IS NULL;

SELECT *
FROM timestamps -- our „for-Loop“
WHERE NOT EXISTS (
SELECT *
FROM temperatures
WHERE temperatures.datetime
= timestamps.datetime
);

Beat Vontobel
CTO, MeteoNews AG

Finding missing rows
WHERE temperatures.stat_id IS NULL
timestamps temperatures
datetime stat_id datetime tt
… … … …
2004-11-03 23:00:00 ABO 2004-11-03 23:00:00 9.9
2004-11-03 23:10:00 ABO 2004-11-03 23:10:00 7.8
2004-11-03 23:20:00 NULL NULL NULL
2004-11-03 23:30:00 NULL NULL NULL
2004-11-03 23:40:00 NULL NULL NULL
2004-11-03 23:50:00 NULL NULL NULL
2004-11-04 00:00:00 NULL NULL NULL
2004-11-04 00:10:00 ABO 2004-11-04 00:10:00 9.2
… … … …

Beat Vontobel
CTO, MeteoNews AG

Filling sequence tables: „Manually“
INSERT from an external loop (or a stored procedure)
•

„explode“ a few rows using CROSS JOINs
•

INSERT INTO i VALUES (1), (2), …, (8), (9), (10);
‣

INSERT INTO j
‣
SELECT u.i * 10 + v.i
FROM i AS u
CROSS JOIN i AS v;
„Pop quiz: generate 1 million records“ (Giuseppe Maxia)
•
http://datacharmer.blogspot.com/2007/12/pop-quiz-generate-1-
million-records.html

Beat Vontobel
CTO, MeteoNews AG

…or, just use the SeqEngine

-- http://seqengine.org
-- README for build instructions

INSTALL PLUGIN SeqEngine SONAME 'ha_seqengine.so';

SHOW PLUGIN; SHOW ENGINES;

CREATE TABLE million (i TIMESTAMP NOT NULL)
ENGINE=SeqEngine CONNECTION=‘1;1000000;1‘;

-- If you want to… now it‘s materialized (fast!)
ALTER TABLE million ENGINE=MyISAM;

Beat Vontobel
CTO, MeteoNews AG

Syntax
-- Variable parts are highlighted

CREATE TABLE table_name (
column_name {INT|TIMESTAMP} NOT NULL [PRIMARY KEY]
) ENGINE=SeqEngine CONNECTION=‘start;end;increment‘;

Beat Vontobel
CTO, MeteoNews AG

„Manually“ created: Disadvantages
Wastes storage
•

Wastes RAM (for caches or if ENGINE=MEMORY)
•

Wastes I/O
•

Wastes CPU (unnecessary overhead in code)
•

Cumbersome to fill (especially if large)
•

Beat Vontobel
CTO, MeteoNews AG

SeqEngine: Disadvantages
None
•

…other than:
•

It‘s (yet) just a really quick hack for this presentation
‣

Contains ugly code and probably a lot of bugs
‣

Coded in C++ by somebody who‘s never done C++
‣
before
Is not part of the core server – go build it yourself!
‣

Beat Vontobel
CTO, MeteoNews AG

Limitations of the SeqEngine (v0.1)
Not real limitations, but due to the concept:
•

Read-only
‣

One column maximum
‣

UNIQUE keys only
‣

Current limitations:
•

INT and TIMESTAMP only
‣

Only full key reads
‣

Error checking, clean-up, optimization, bugs…
‣

Beat Vontobel
CTO, MeteoNews AG

The tiny core of the SeqEngine: Init
int ha_seqengine::rnd_init(bool scan)
{
DBUG_ENTER(quot;ha_seqengine::rnd_initquot;);

rnd_cursor_pos = share->seq_def.seq_start;

DBUG_RETURN(0);
}

Beat Vontobel
CTO, MeteoNews AG

The tiny core of the SeqEngine: Next row
int ha_seqengine::rnd_next(uchar *buf)
{
DBUG_ENTER(quot;ha_seqengine::rnd_nextquot;);

if(rnd_cursor_pos <= share->seq_def.seq_end) {
build_row(buf, rnd_cursor_pos);
rnd_cursor_pos += share->seq_def.seq_inc;
table->status= 0;
DBUG_RETURN(0);
}

table->status= STATUS_NOT_FOUND;
DBUG_RETURN(HA_ERR_END_OF_FILE);
}

Beat Vontobel
CTO, MeteoNews AG

SeqEngine: The BOF
Using the Storage Engine API for small projects
•

Additional questions/discussion
•

Wednesday, April 22
‣

20:30pm
‣

Ballroom E
‣

Beat Vontobel
CTO, MeteoNews AG

Back to the missing rows example…
WHERE temperatures.stat_id IS NULL
timestamps temperatures
datetime stat_id datetime tt
… … … …
2004-11-03 23:00:00 ABO 2004-11-03 23:00:00 9.9
2004-11-03 23:10:00 ABO 2004-11-03 23:10:00 7.8
2004-11-03 23:20:00 NULL NULL NULL
2004-11-03 23:30:00 NULL NULL NULL
2004-11-03 23:40:00 NULL NULL NULL
2004-11-03 23:50:00 NULL NULL NULL
2004-11-04 00:00:00 NULL NULL NULL
2004-11-04 00:10:00 ABO 2004-11-04 00:10:00 9.2
… … … …

Beat Vontobel
CTO, MeteoNews AG

SeqEngine (LEFT JOIN)
SELECT timestamps.datetime, stations.stat_id

FROM timestamps CROSS JOIN stations

LEFT JOIN temperatures AS temps
ON (temps.datetime, temps.stat_id)
= (timestamps.datetime, stations.stat_id)

WHERE stations.stat_id = 'ABO'
AND temperatures.stat_id IS NULL;

Beat Vontobel
CTO, MeteoNews AG

SeqEngine (NOT EXISTS)
SELECT timestamps.datetime, stations.stat_id

FROM timestamps CROSS JOIN stations

WHERE stations.stat_id = 'ABO'

AND NOT EXISTS (
SELECT *
FROM temperatures AS temps
WHERE (temps.datetime, temps.stat_id)
= (timestamps.datetime, stations.stat_id)
);

Beat Vontobel
CTO, MeteoNews AG

As a Procedure (Single SELECTs)
CREATE PROCEDURE find_holes_naive(stat CHAR(3))
BEGIN
DECLARE dt DATETIME DEFAULT '2004-01-01 00:00:00';
DECLARE c INT;

WHILE dt < '2005-01-01 00:00:00' DO
SELECT COUNT(*) INTO c
FROM temperatures
WHERE (stat_id, datetime) = (stat, dt);

IF c = 0 THEN -- missing row
SELECT stat, dt;
END IF;

SET dt = dt + INTERVAL 10 MINUTE;
END WHILE;
END //
Beat Vontobel
CTO, MeteoNews AG

As a Procedure (Ordered Set)
CREATE PROCEDURE find_holes_ordered(stat CHAR(3))
BEGIN
DECLARE no_more_rows BOOLEAN DEFAULT FALSE;
DECLARE dt1 DATETIME DEFAULT '2004-01-01 00:00:00';
DECLARE dt2 DATETIME;

DECLARE temperatures_cursor CURSOR FOR
SELECT datetime FROM temperatures WHERE stat_id = stat ORDER BY datetime ASC;

DECLARE CONTINUE HANDLER FOR NOT FOUND SET no_more_rows = TRUE;

OPEN temperatures_cursor;

temperatures_loop: LOOP
FETCH temperatures_cursor INTO dt2;

WHILE dt1 != dt2 DO
SELECT stat, dt1;
SET dt1 = dt1 + INTERVAL 10 MINUTE;
IF dt1 >= '2005-01-01 00:00:00' THEN LEAVE temperatures_loop; END IF;
END WHILE;

SET dt1 = dt1 + INTERVAL 10 MINUTE;
IF dt1 >= '2005-01-01 00:00:00' THEN LEAVE temperatures_loop; END IF;

END LOOP temperatures_loop;

CLOSE temperatures_cursor;
END//

Beat Vontobel
CTO, MeteoNews AG

Self-Reference (LEFT self-JOIN)
SELECT *
FROM temperatures

LEFT JOIN temperatures AS missing

ON temperatures.stat_id = missing.stat_id
AND temperatures.datetime + INTERVAL 10 MINUTE
= missing.datetime

WHERE temperatures.stat_id = 'ABO'
AND missing.datetime IS NULL;

Beat Vontobel
CTO, MeteoNews AG

Self-Reference (NOT EXISTS)
SELECT *
FROM temperatures

WHERE NOT EXISTS (

SELECT *
FROM temperatures AS missing
WHERE missing.datetime
= temperatures.datetime
+ INTERVAL 10 MINUTE
AND missing.stat_id
= temperatures.stat_id

)

AND stat_id = 'ABO';

Beat Vontobel
CTO, MeteoNews AG

What‘s the performance?
SeqEngine
•

LEFT JOIN
‣

NOT EXISTS
‣

Self-Reference
•

LEFT self-JOIN
‣

NOT EXISTS
‣

Stored Procedures
•

Naïve (Single SELECTs)
‣

Standard (Ordered SET)
‣
Beat Vontobel
CTO, MeteoNews AG

The benchmark
All the usual disclaimers for benchmarks apply: Go ahead and measure it with your hardware, your version of MySQL,
your storage engines, your data sets and your server conﬁguration settings.

Query Remarks Time [s]
1 SeqEngine (NOT EXISTS) 0.28

2 SeqEngine (LEFT JOIN) 0.29

3 Procedure (Ordered SET) result set per missing row 0.59

4 Self (NOT EXISTS) only first missing row 0.93

5 Self (LEFT JOIN) only first missing row 1.10

6 Procedure (Single SELECTs) result set per missing row 2.80

Beat Vontobel
CTO, MeteoNews AG

The benchmark
All the usual disclaimers for benchmarks apply: Go ahead and measure it with your hardware, your version of MySQL,
your storage engines, your data sets and your server conﬁguration settings.

1. SeqEngine (NOT EXISTS) 0.28s

2. SeqEngine (LEFT JOIN) 0.29s

3. Procedure (Ordered SET) 0.59s

4. Self Reference (NOT EXISTS) 0.93s

5. Self Reference (LEFT JOIN) 1.10s

6. Procedure (Single SELECTs) 2.80s

0s 0.5s 1.0s 1.5s 2.0s 2.5s 3.0s

Beat Vontobel
CTO, MeteoNews AG

Lessons to be learned…
The Sequence trick (and SeqEngine) worked
•

It may sometimes pay off to go the extra mile and write a
‣
custom storage engine!
Stored PROCEDUREs with CURSORs sometimes
•
can be damned fast!
Subquery optimization really did progress in MySQL
•
(at least in some parts, more to come with 6.0)
Consider NOT EXISTS over LEFT JOIN
‣

Beat Vontobel
CTO, MeteoNews AG

2nd use case: Generate Test Data
mysql> CREATE TABLE large (i INT NOT NULL)
ENGINE=SeqEngine CONNECTION='1;10000000;1';

Query OK, 0 rows affected (0,12 sec)

mysql> ALTER TABLE large ENGINE=MyISAM;

Query OK, 10000000 rows affected (3,27 sec)
Records: 10000000 Duplicates: 0 Warnings: 0

Beat Vontobel
CTO, MeteoNews AG

Generating other Sequences from Integers
CREATE VIEW letters AS
SELECT CHAR(i) FROM integer_sequence;

CREATE VIEW timestamps AS
SELECT FROM_UNIXTIME(i) FROM integer_sequence;

CREATE VIEW squares AS
SELECT i*i FROM integer_sequence;

…

Beat Vontobel
CTO, MeteoNews AG

Generate very large and complex data sets
INSERT INTO customers
SELECT i AS customer_id,
MD5(i) AS customer_name,
ROUND(RAND()*80+1920) AS customer_year
FROM large;

SELECT * FROM customers;
+-------------+---------------------+---------------+
| customer_id | customer_name | customer_year |
+-------------+---------------------+---------------+
| 1 | c4ca4238a0b9f75849… | 1935 |
| 2 | c81e728d9d4c2f636f… | 1967 |
| |
| |
| …|… | …|
+-------------+---------------------+---------------+
10000000 rows in set
Beat Vontobel
CTO, MeteoNews AG

„Salvage“ a bad design
CREATE TABLE salvage (
col INT NOT NULL
) ENGINE=SeqEngine CONNECTION='1;3;1';

+-----+
| col |
+-----+
| 1|
| 2|
| 3|
+-----+

Beat Vontobel
CTO, MeteoNews AG

Querying Comma-Separated Attribute Lists
SELECT username,

SUBSTRING_INDEX(

SUBSTRING_INDEX(

selections,
',', i
),

',', -1
) AS selection
FROM selections
JOIN integers
HAVING selection NOT LIKE '';

Beat Vontobel
CTO, MeteoNews AG

Querying Comma-Separated Attribute Lists
SELECT username,
-- Take last element
SUBSTRING_INDEX(

-- Crop list after element i
SUBSTRING_INDEX(
-- Add empty sentinel element
CONCAT(selections, ','),
',', i
),

',', -1
) AS selection
FROM selections
JOIN integers

Beat Vontobel
CTO, MeteoNews AG

Problem: Variable-sized IN-Predicates
Statements can‘t be prepared for variable-sized lists in the
•
in clause:
SELECT * FROM x WHERE a IN (?)
‣

One needs:
•

SELECT * FROM x WHERE a IN (?)
‣

SELECT * FROM x WHERE a IN (?, ?)
‣

SELECT * FROM x WHERE a IN (?, ?, ?, …)
‣

Example from Stéphane Faroult: „The Art of SQL“
•
adapted for MySQL

Beat Vontobel
CTO, MeteoNews AG

Split arguments as before!
SELECT …
FROM rental
INNER JOIN customer ON rental.customer_id = …
INNER JOIN address ON …
…
INNER JOIN (
SELECT SUBSTRING_INDEX(
SUBSTRING_INDEX(CONCAT(?, quot;,quot;), quot;,quot;, i),
quot;,quot;, -1
) AS customer_id
FROM sequences.integers
WHERE i <= ?
) AS s ON rental.customer_id = s.customer_id
…
WHERE …;

Beat Vontobel
CTO, MeteoNews AG

SQL-String-Parsing beats Query-Parsing!
Execution Times in Seconds for a different number of runs (lower is better)
20
18.1

16
16.3
12.1
12
10.9
8
6.0

4 5.4
0.6
0
0.5
x

0x

0x

0x
00

00

00

00
10

10

20

30
Prepared/Sequence Client-side IN-List

Beat Vontobel
CTO, MeteoNews AG

Sequences and SeqEngine: Conclusion
Use Sequences (and SeqEngine) to e.g.:
•

Find missing rows
‣

Generate test data
‣

Pivot tables
‣

Do clever-things with „for-Loops“ (String-Parsing etc.)
‣

•

Slides will be available shortly after the presentation
‣
(also on conference website)

Beat Vontobel
CTO, MeteoNews AG

Solving Common Sql Problems With The Seq Engine

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Mehr von MySQLConference

Mehr von MySQLConference (16)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Solving Common Sql Problems With The Seq Engine