The document discusses the MariaDB CONNECT storage engine, which allows querying external file formats from within MariaDB. It was created by database expert Olivier Bertrand and brings business intelligence capabilities to MariaDB by enabling access to data sources like CSV, XML, Excel and other formats without needing ETL processes. The storage engine uses the MySQL plugin architecture and implements features like indexing, condition pushdown, and support for ODBC, MySQL tables, and various file types.

1. SkySQL
MariaDB
CONNECT Storage Engine
Serge Frezefond
SkySQL Ab 2012 Confidential

2. SkySQL Ab 2012 Confidential
Goal of the CONNECT Storage Engine :
BI on various file formats
Most of the data in companies is in various
external datasources (many in non relational
database format) :
– Dbase, Firebird, SQlite
– csv,
– xml,
– stored per column...
– Microsoft Access & Excel
– Distributed mysql servers
Targeting BI data access on these formats.
Not targeted for OLTP

3. SkySQL Ab 2012 Confidential
Behind the scene
Traditional BI
Data is processed by an ETL
– Change in the data model(denormalization...)
Agregates are computed
– Need to be defined and maintained
Might need to move data out of RDBMS to
other kind of datastore
– OLAP, Collumn store, Hadoop/Hbase ...
Specific tools are used to query the data
IT is involved to maintain this machinery

4. SkySQL Ab 2012 Confidential
MariaDB CONNECT Storage Engine :
created by Olivier Bertrand
IBM database researcher
– Now retired, 50 years expertise programing
Very experienced on databases
– Worked on system-R, DB2, natural language
query ...
Discovered MySQL when looking for friendly
place to test new concepts.(2004)
– Decided to go open source
– Started to appreciate the MariaDB openess
and friendlyness

5. SkySQL Ab 2012 Confidential
How did the CONNECT Storage Engine
move to MariaDB?
Olivier met Monty creator of MySQL and MariaDB a
few years ago (2004 for other concepts)
SkySQL met Olivier and we start working together :
– First access to launchpad, go to linux, test,
Olivier start working with MariaDB team :
– Testing, bug fixes, security, test cases ...
SkySQL / MariaDB and Olivier agreed that is was
ready to be released and supported under GPL
– MariaDB flexibility ease integration

6. SkySQL Ab 2012 Confidential
The CONNECT Storage Engine
Uses the MySQL Plugin Architecture
A major differentiator of MySQL
Allow very specific datastore to interact with
the MySQL sql layer
Allow advanced interaction
– Specific Create Table parameters(MariaDB)
– Condition push down
Allow join with other storage engines
– InnoDB / MyISAM tables

7. SkySQL Ab 2012 Confidential
The CONNECT Storage Engine
implements advanced features
●
Support of external data sources :
–
Odbc, MySQL, WMI ...
●
Support multi files tables
●
Support Big File Table > 2G
●
Support virtual tables (DIR)
●
Add autocreate of tables :
–
The structure is discovered from the data
source
●
Use MariaDB create table new parameters
capability (avoid comments polution)
●
Support compressed tables

8. SkySQL Ab 2012 Confidential
The CONNECT Storage Engine
implements advanced features
●
Add indexing to files
– index optimized for read
●
Condition Push down
– Used with ODBC and MySQL to push condition
to the target database. Big perf gain.
●
Support MariaDB virtuals columns
●
Support of special columns :
– Rowid, fileid, tabid, servid
●
Muti tables table (like merge)
– Different structure, not myisam only, remotely
distributed tables

9. SkySQL Ab 2012 Confidential
CONNECT Storage Engine vs.
Federated(X) storage Engine
- support ODBC
– With muti files ODBC
- support access to MySQL table
- support the limit clause

10. SkySQL Ab 2012 Confidential
CONNECT Storage Engine
ODBC component
Allow to access to any datasource accessible
through ODBC.
– Excel
– Access
– Firebird
– SQLite
– SQL Server, Oracle, DB2
– ...
Possibility to do multifiles ODBC
– To query consolidated monthly excel datasheet

11. SkySQL Ab 2012 Confidential
The CONNECT Storage Engine
implements advanced features
Catalog table :
– For Example Describe for odbc table
– No need to do create table
– Access to data / column metadata
Memory file maping
– For file type table (not xml)
Table format .ini

12. SkySQL Ab 2012 Confidential
The CONNECT Storage Engine
implements advanced features
Multiple CONNECT tables can be created on the
same underlying file
– Indexes can be shared between tables

13. SkySQL Ab 2012 Confidential
Connect Storage Engine
VEC table / Column store
col1col1
col2col1
col3col2col1
row3row2row1
col3
col1
freefree
freefreefree
col3
free
-1 or per column file
- Indexes work
- Fixed size record

14. SkySQL Ab 2012 Confidential
Connect Storage Engine
Table List Table (// Merge)
col1col1 col2
col1 col2
col3
col3col1 col2
col3col1 col2
ODBC table
MySQL table
col4
Muti tables table (like merge)
– Different structure, not myisam
only, remotely distributed tables

15. SkySQL Ab 2012 Confidential
Connect Storage Engine vs.
MySQL Merge tables
Table list table :
- support non MyISAM tables
- no need to the exact same structure for table
- underlying tables can be remote
– Distributed architecture

16. SkySQL Ab 2012 Confidential
Importing /exporting MySQL data
in various formats
Importing file data into MySQL tables
– Here for example from an XML file :
• create table biblio select * from xsampall2;
Exporting data from MySQL
– Here for example we export to XML format :
create table handout engine=CONNECT table_type=XML
file_name='handout.htm' header=yes
option_list='name=TABLE,coltype=HTML,attribute=border=1;cellpadding=5'
select plugin_name handler, plugin_version version, plugin_author
author, plugin_description description, plugin_maturity maturity
from information_schema.plugins where plugin_type = 'STORAGE ENGINE';

17. SkySQL Ab 2012 Confidential
Where is the MariaDB Connect Storage
Engine available ?
●
It is 100 % open source
●
It is available on MariaDB launchpad
●
Open Bug database
●
Public Roadmap
●
Released test cases
●
Improvement request / worklog

18. SkySQL Ab 2012 Confidential
How you can help
Adopt it / Test it.
Bugs : report bugs / propose fixes
Documentation : help improve it
Sharing : test it, blog about it,
– Share your experience about interesting
usages.

19. SkySQL Ab 2012 Confidential
Conclusion
●
The MariaDB Connect Storage Engine :
●
Open MariaDB to BI and data analysis
●
Brings real value to MariaDB users
●
Illustrates openess of MariaDB community
●
Supported by SkySQL / MariaDB

20. SkySQL Ab 2012 Confidential
20
Thank You
Serge Frezefond
www.skysql.coml
www.facebook.com/skysql
www.linkedin.com/company/skysql

21. SkySQL Ab 2012 Confidential
Enhancements considered
for MariaDB connect storage engine
- parallel access to multitables table
-