MySQL Data Warehousing Seminar London Guide
•
3 gefällt mir•1,521 views
This document summarizes a breakfast seminar on data warehousing solutions with MySQL that was held in London on February 4th, 2010. The agenda included introductions, presentations on using MySQL for data warehousing, Infobright, Talend, and time for questions. The seminar provided an overview of MySQL's data warehousing strategy and ecosystem, common use cases, storage engines, and best practices for modeling, queries, partitioning, replication and other technologies.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (8)
Ähnlich wie MySQL Data Warehousing Seminar London Guide
Ähnlich wie MySQL Data Warehousing Seminar London Guide (20)
Mehr von Ivan Zoratti
Mehr von Ivan Zoratti (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
MySQL Data Warehousing Seminar London Guide
- 1. Data Warehousing Solutions with MySQL A Breakfast Seminar in London 4th Feb 2010 1 Sunday, 7 February 2010
- 2. 9:00 - Welcome Coffee and Tea 9:20 - Introduction 9:30 - MySQL for Data Warehousing 10:00 - Infobright 10:30 - Coffee/Tea Break 10:45 - Talend 11:30 - Seminar Ends. 2 Sunday, 7 February 2010
- 3. Introduction Sunday, 7 February 2010
- 4. MySQL Market Segments ` Web / Web 2.0 OEM / ISV's On Demand, SaaS, Hosting Telecommunications Enterprise 2.0 Open-Source Powers the Web & The Network 4 Sunday, 7 February 2010
- 5. Timeline MAR 2008 Sun acquired MySQL completed March 2008 Good acquisition, MySQL continues to grow APR 2009 April 2009 : ORCL agreement to acquire Sun JAN 2010 The EC gives full clearance to the acquisition FEB 2010 We continue to develop, maintain, market, sell and support MySQL! 5 Sunday, 7 February 2010
- 6. Oracle’s MySQL Strategy • Becomes part of the Open Source GBU > Independent sales organisation - retained from Sun > Independent development organisation – retained from Sun • Make MySQL better > Apply Oracle’s expertise and engineering processes > A natural extension of what Oracle has done with InnoDB • Make MySQL support better > Leverage Oracle’s award winning global support infrastructure • Make MySQL part of the Oracle stack > Many customers use both MySQL and Oracle database > Integrate with Enterprise Manager, Secure Backup, Audit Vault http://www.oracle.com/ocom/groups/public/@ocom/documents/webcontent/044521.pdf 6 Sunday, 7 February 2010
- 7. Enjoy the event! 7 Sunday, 7 February 2010
- 8. Data Warehousing with MySQL Sunday, 7 February 2010
- 9. MySQL Data Warehousing Strategy • Strongly support common data warehouse use cases • Offer modern technology that adheres to MySQL’s software priorities (reliability, performance, ease-of-use) • Partner with major BI/ETL vendors • Offer highly attractive total cost of ownership 9 Sunday, 7 February 2010
- 10. The MySQL DW Ecosystem BI/REPORTING ETL INTEGRATION TOOLS RDBMS STORAGE ENGINE PLATFORM 10 Sunday, 7 February 2010
- 11. Common Use Cases 1.Small, semi real-time data marts 2.Continuous, real-time/query data warehousing 3.Traditional, standard reporting warehouse 4.Massive historical, with ad-hoc queries warehouse 5.BI, analytic in OLTP applications (emerging…) Data Mart Real-Time Traditional Historical Analytical SQL 11 Sunday, 7 February 2010
- 12. MySQL Technical Strategy • Provide open source architecture to maximize innovation • Offer core data warehousing feature set • Provide specialised data warehouse engines for key use cases • Supply strategies for combating mixed workload challenge 12 Sunday, 7 February 2010
- 13. Pluggable Storage Engine Architecture 13 Sunday, 7 February 2010
- 14. MySQL Enterprise • MySQL Enterprise Server • Monthly Rapid Updates Server • Quarterly Service Packs • Hot Fix Program • Indemnification • Global Monitoring of All Servers • Web-Based Central Console Monitor • Built-in Advisors and Expert Advice • MySQL Query Analyzer • Replication Monitor • 24 x 7 x 365 Production Support • Web-Based Knowledge Base Support • Consultative Help • High Availability and Scale Out http://www.mysql.com/products/enterprise/ 14 Sunday, 7 February 2010
- 15. MySQL Enterprise Monitor “Your Virtual MySQL DBA” Assistant • Single, consolidated view into entire MySQL environment • Auto discovery of MySQL Servers, Replication Topologies • New Query Analyzer • Customisable rules-based monitoring and alerts • Identifies problems before they occur • Reduces risk of downtime • Makes it easier to scale-out without requiring more DBAs http://www.mysql.com/products/enterprise/advisors.html 15 Sunday, 7 February 2010
- 16. MySQL Query Analyzer • Centralised monitoring of Queries across all servers • No reliance on Slow Query Logs, SHOW PROCESSLIST, VMSTAT, etc. • Aggregated view of query execution counts, time, and rows • Saves time parsing atomic executions for total query expense “Finds code problems before your customers do.” 16 Sunday, 7 February 2010
- 17. The MySQL Technology behind a DW Strategy SHARDING REPLICATION MySQL PROXY MEMCACHED QUERY CACHE STORAGE PARTITIONING ENGINES Col1 Col2 Col3 Col4 Col5 Col1 Col2 Col3 Col4 Col5 Col1 Col2 Col3 Col4 Col5 17 Sunday, 7 February 2010
- 18. Warehouse use cases/mapping Data Mart Real-Time Traditional Historical Analytical SQL •MyISAM •MyISAM •MyISAM •MyISAM •MyISAM •InnoDB •InnoDB •InnoDB •InnoDB •InnoDB •CSV •CSV •CSV •CSV •CSV •Archive •Archive •Archive •Archive •Archive •Federated •Federated •Federated •Federated •Federated •Query Cache •Query Cache •Query Cache •Query Cache •Query Cache •Replication •Replication •Replication •Replication •Replication •Sharding •Sharding •Sharding •Sharding •Sharding •Proxy •Proxy •Proxy •Proxy •Proxy •Memcached •Memcached •Memcached •Memcached •Memcached 18 Sunday, 7 February 2010
- 19. MySQL Data Warehouse Cookbook Sunday, 7 February 2010
- 20. Partitioning • Partition Pruning • Partitioning key must result in an INT • Check table lock with MyISAM • Check the number of open files • Foreign Keys, Fulltext and spatial indexes are not supported • No MyISAM, LOAD INDEX or INSERT DELAYED • For DW, it is mainly limited to InnoDB and MyISAM Vertical Partitioning Horizontal Partitioning Col1 Col2 Col3 Col4 Col5 Col1 Col2 Col1 Col3 Col4 Col5 Col1 Col2 Col3 Col4 Col5 Col1 Col2 Col3 Col4 Col5 Col1 Col2 Col3 Col4 Col5 20 Sunday, 7 February 2010
- 21. SQL Generation • Multipass SQL or Subqueries • Avoid complex queries > More efficient use of query cache, key buffer and buffer pool > More shard friendly > More scalable for the current version of MySQL –No parallel query • Use temp tables and stored procedures • Check with EXPLAIN > ALL (sequential scan) > Using filesort > Using temporary (for GROUP BY and ORDER BY) 21 Sunday, 7 February 2010
- 22. Server Tuning Query Cache Temporary Tables • SELECT...SQL_NO_CACHE • tmp_table_size • query_cache_type • max_heap_table_size • query_cache_limit • Implicit tmp tables can be tricky to control • query_cache_size • Store intermediate results • No time functions • Connect > Query > Disconnect Thread Buffers • join_buffer_size • read_buffer_size • read_rnd_buffer_size • sort_buffer_size • For large resultsets and for high number of concurrent users, they should be set individually or by role 22 Sunday, 7 February 2010
- 23. Modelling • Multidimensional, but with care • Queries • Snowflake vs Star Schema > Query on Dimension N > Temp Table > Do not denormalise descriptions > Query on Fact 1 > Temp Table > Multiple fact tables with 1:1 relationships > Query on Fact 2 Join Temp Table Key Desc Key Desc Key Desc Key Desc Key Desc Key Desc Key Desc Key Desc Key Key Desc Key Key Desc Key Key Key Desc Key Key Key Desc PK Key Key Key Key Met Met Met Met Met PK Key Key Key Key Met Met Met Met Met Key Key Key Desc Key Key Key Desc Key Desc Key Desc Key Desc Key Desc Key Desc Key Desc Key Key Desc Key Key Desc Key Desc Key Desc PK Key Key Key Key ... Key Met Met Met PK Met Met Met Met Met Met Met 23 Sunday, 7 February 2010
- 24. Storage Engines MyISAM CSV • Compressed Tables • Good ETL trick • Use different spindles for data and indexes • No Partitioning, no indexing, no nulls • Fast inserts - Insert already sorted data (when possible) • Key Buffers • Multiple Key Buffers • SET GLOBAL <key_cache_name>.key_buffer_size... Archive • CACHE INDEX ... IN ... • Data compression and fast retrieve • key_cache_block_size • INSERT & SELECT • bulk_insert_buffer_size • No index (autoincrement only) • Spatial and Fulltext indexes • All active shared disk cluster Federated InnoDB • Limited indexing • innodb_file_per_table • Tips: • innodb_flush_log_at_trx_commit • Queries can be executed on multiple servers + result • innodb_buffer_pool_size collection • The new Innodb plugin • Use of stored procedures to consolidate results and control the access to the FEDERATED tables • Fast index creation • Data compression • Do not use FK or constraints 24 Sunday, 7 February 2010
- 25. Replication Source Master • [For some] The easiest way to provide real time data marts Querying Updating • Tips: Rotating > Delayed replication Slaves > Rotating servers > Support to more power users BI/Report Read Servers Write Real -10 -30 -1 -12 Yesterday Time Min Min Hour Hours Source Master 25 Sunday, 7 February 2010
- 26. Sharding • Sharding > Great to distribute the workload > Fantastic if the queries can be executed in parallel thanks to a middle or a client layer > Tips: – Replicate the dimensions – specialise shards on facts – partition facts on shards BI/Report Read Servers Write Shards A1 A2 B C1 C2 D Dimensions Master 26 Sunday, 7 February 2010
- 27. More Resources Available • Webinars • http://www-it.mysql.com/news-and-events/web-seminars/ • Consulting • MySQL Architecture & Design • MySQL Performance tuning http://www.mysql.com/consulting/ • Training • MySQL 5.1 for developers • MySQL 5.1 for DBAs http://www.mysql.com/training/ • White Papers • http://www.mysql.com/why-mysql/white-papers/ 27 Sunday, 7 February 2010
- 28. Thank You! Data Warehouse Solutions with MySQL ivan@mysql.com http://izoratti.blogspot.com 28 Sunday, 7 February 2010