Polyglot Persistence for the MongoDB, PostgreSQL & MySQL DBA
The introduction of DevOps in organisations has changed the development process, and perhaps introduced some challenges. Developers, in addition to their own preferred programming languages, also have their own preference for backend storage.The former is often referred to as polyglot languages and the latter as polyglot persistence.
Having multiple storage backends means your organization will become more agile on the development side and allows choice to the developers but it also imposes additional knowledge on the operations side. Extending your infrastructure from only MySQL, to deploying other storage backends like MongoDB and PostgreSQL, implies you have to also monitor, manage and scale them. As every storage backend excels at different use cases, this also means you have to reinvent the wheel for every one of them.
This webinar covers the four major operational challenges for MySQL, MongoDB & PostgreSQL:
Deployment
Management
Monitoring
Scaling
And how to deal with them
SPEAKER
Art van Scheppingen is a Senior Support Engineer at Severalnines. He’s a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.
This webinar is based upon the experience Art had while writing our How to become a ClusterControl DBA blog series and implementing multiple storage backends to ClusterControl. To view all the blogs of the ‘Become a ClusterControl DBA’ series visit: http://severalnines.com/blog-categories/clustercontrol
2. Confidential
Logistics
Webinar is recorded
Replay available soon
Feel free to ask questions at any time
Use your control panel to contact us
Or email us as well: info@severalnines.com
Copyright Severalnines AB
2
3. Confidential
Agenda
What is Polyglot Persistence?
Example project
Is Polyglot Persistence good or bad?
Leverage ClusterControl for Polyglot Persistence (demo)
Q&A
3
Copyright Severalnines AB
6. Confidential
Where does the term come from?
The way we work is changing
Enterprise applications are becoming more complex
Separate (devops/agile) teams
Ownership of applications
(Micro)services
Everyone has their preference
Various programming languages
Various storage systems
Copyright Severalnines AB
6
7. Confidential
Where does the term come from?
Monoglot Programming
Only one programming language allowed
Readability
All code is in the same language
Support
One platform to support
Knowledge
Everybody is an expert
Is there a jack-of-all-trades language?
Copyright Severalnines AB
7
10. Confidential
Polyglot Programming
Polyglot Programming
Use programming languages for what they are good at
Flexibility
Use Java for a secure API
Use Scala for real time stream processing
Use Python for text analysis
Tie everything together using AngularJS
Knowledge
Everybody is expert at one or more languages
Copyright Severalnines AB
10
11. Confidential
Polyglot Programming
Copyright Severalnines AB
11
User
Website using
micro services
Profile API Billing API Recommendations API
login
microservice
Java
profile details
microservice
Python
billing
microservice
Go
recommendations
microservice
Scala
12. Confidential
Monoglot Persistence
Copyright Severalnines AB
12
User
Website using
micro services
Profile API Billing API Recommendations API
login
microservice
Java
profile details
microservice
Python
billing
microservice
Go
recommendations
microservice
Scala
MySQL
13. Confidential
Data storage landscape changes
Relational data stores (RDBMS)
Key-Value data stores
Columnar data stores
Document data stores
Graph data stores
Copyright Severalnines AB
13
17. Confidential
Polyglot Persistence
Complex problems require different storage systems
Use the right tool for the job, for example
Use PostgreSQL for financial data
Use MySQL for website contents
Use MongoDB for user profiles
Use Cassandra for real time streams
Use Neo4J for recommendation analysis
Copyright Severalnines AB
17
18. Confidential
Use the right tool for the right job
Document storage: MongoDB
Copyright Severalnines AB
18
application
+ mongos
MongoDB
replica set
shard1
objectsshard
1
{
_id: <objectid1>,
username: myuser,
phone: 023 443 2234,
email: johndoe@gmail.com
}
{
_id: <objectid2>,
username: anotheruser,
email: anotheruser@gmail.com,
age: 23,
}
MongoDB
replica set
shard2
replication
objectsshard2
19. Confidential
Use the right tool for the right job
Columnar storage: Cassandra
Copyright Severalnines AB
19
21. Confidential
Polyglot Persistence
Copyright Severalnines AB
21
User
Website using
micro services
Profile API Billing API Recommendations API
login
microservice
Java
profile details
microservice
Python
billing
microservice
Go
recommendations
microservice
Scala
PostgreSQL Hadoop RedisMongoDBMySQL
23. Confidential
Project management lifecycle
Copyright Severalnines AB
23
Initiating phase Planning phase Executing phase Closing phase
Requirements
Analysis
Design Develop Test Implement
Handover to
Ops
Systems
decomission
24. Confidential
Project management lifecycle
Copyright Severalnines AB
24
Initiating phase Planning phase Executing phase Closing phase
Requirements
Analysis
Design Develop Test Implement
Handover to
Ops
Systems
decomission
25. Confidential
Project management lifecycle
Copyright Severalnines AB
25
Initiating phase Planning phase Executing phase Closing phase
Requirements
Analysis
Design Develop Test Implement
Handover to
Ops
Systems
decomission
26. Confidential
Development
Requirements are gathered
Create a recommendation system
Storage must be (blazingly) fast
Document must contain user details
Data must be kept until the next visit
Choose the right tool for the right job
Store data as a document
Use MongoDB
Development
Copyright Severalnines AB
26
27. Confidential
But what happens after development?
Delivery
Ops sets up the application on the infrastructure
Monitoring is added to see if the application works
If everything is fine the product will be handed over
What’s next?
Systems monitoring
Backups
Scaling
Software updates
BI
Copyright Severalnines AB
27
28. Confidential
The ops perspective
Copyright Severalnines AB
28
Initiating
phase
Planning
phase
Executing
phase Closing phase
Development
Handover to
Ops
Involvement
Maintenance: keep systems
healthy
Systems
decomission
29. Confidential
Systems monitoring
Supported data stores so far:
MySQL
Memcached
How do we monitor MongoDB?
Integration with Nagios / Zabbix / Shinken
Integration with Graphite / OpenTSDB / InfluxDB
What should we monitor?
Key performance metrics
SQL vs Mongo queries
Copyright Severalnines AB
29
30. Confidential
Backups
Supported backups so far
Xtrabackup streams
How do we backup MongoDB?
What should we backup from MongoDB?
How do we restore from a backup?
How do we provide a backup for development?
Copyright Severalnines AB
30
31. Confidential
Scaling out
We are data hoarders and like to keep everything
Scaling options in MySQL
Read-bound workloads: add more slaves
Write-bound workloads: sharding or NDB Cluster
Scaling in MongoDB?
When should we scale out?
How do we add a new shard?
How do we redistribute between shards?
How do we add a new read slave?
Copyright Severalnines AB
31
32. Confidential
Software updates
When do we need to update with MySQL?
Once we hit a bug?
Every major release?
Every minor release?
When do we need to update with MongoDB?
How do we know the impact?
Copyright Severalnines AB
32
33. Confidential
Business Intelligence
BI with MySQL?
Export full database from a slave
Allow schemas/tables to be queried for specific tables
Most BI tools can connect to MySQL
BI with MongoDB?
Native MongoDB analytics tools
Write your own exports
BI connector (since version 3.2)
Copyright Severalnines AB
33
34. Confidential
A new project emerges
Copyright Severalnines AB
34
Initiating phase Planning phase Executing phase Closing phase
Requirements
Analysis
Design Develop Test Implement
Handover to
Ops
Systems
decomission
35. Confidential
With new requirements
Requirements are gathered
Allow users to view their earnings
Data must always be 100% reliable
Storage must be ACID compliant
Choose the right tool for the right job
Store data relational
Use PostgreSQL
Copyright Severalnines AB
35
37. Confidential
The drawbacks
More specific (domain) knowledge is needed
Additional tooling may be necessary
Need more hands to maintain the systems
Copyright Severalnines AB
37
38. Confidential
The benefits
More flexibility to your company
Driven and dedicated devops who take ownership
Use the right tool for the right job
Copyright Severalnines AB
38
39. Confidential
The challenges
Deploy various data stores
Handle HA and failovers
Scaling and sharding
Interpret graphs and monitoring
Schedule backups and handle recovery
Handle updates/upgrades
Copyright Severalnines AB
39
40. Confidential
Polyglot Persistence is here to stay
Variety of data storage needs keeps growing
Polyglots in the cloud
Better be prepared for it
Copyright Severalnines AB
40