SlideShare ist ein Scribd-Unternehmen logo
1 von 72
Downloaden Sie, um offline zu lesen
Polyglot Persistence
      Two Great Tastes
  That Taste Great Together!


                      John Wood
               john_p_wood@yahoo.com
                     @johnpwood
About Me
●   Software Developer at Interactive Mediums
●   Primarily work on a web application that allows
    our customers to engage and interact with their
    customers
●   Writing code for about 15 years
●   Tinkering with NoSQL for about 1.5 years
●   Have a NoSQL solution that has been running
    in production for a year
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
You Now Have A Choice
The RDBMS Is No Longer The
      Default Choice
The RDBMS Is No Longer The
           Default Choice
●   Can be very difficult to scale horizontally
●   Schemas can be difficult to maintain and
    migrate
●   For some applications, the data integrity
    features of the RDBMS are an unnecessary
    overhead
●   Data constraints and JOINs can be expensive
    at runtime
NoSQL Databases Have Stepped
  Up To Address These Issues
NoSQL Databases Have Stepped
      Up To Address These Issues

●   Schema-less
●   Little to no data integrity enforcement
●   Self-contained data
●   Eventually consistent
●   Easy to scale horizontally to add processing
    power and storage
But The RDBMS Is Far From Dead
But The RDBMS Is Far From Dead
●   Incredibly mature, and battle tested
●   Immediate and constant consistency
●   Integrity of data is enforced
●   Efficient use of storage space if data
    normalized properly
●   Supported by everyone and everything (tools,
    frameworks, libraries, etc)
●   Incredibly flexible and powerful query language
●   Help is plentiful and easy to find
Choice is good...right?
Decisions, Decisions...
You Don't Have to
     Choose
“You've got your chocolate in my peanut butter!”
Polyglot Persistence
pol●y●glot - Adjective
Knowing or using several languages
pol●y●glot - Adjective
  Knowing or using several languages



        per●sist●ence - Noun
The continued or prolonged existence of
              something
Polyglot Persistence
The continued or prolonged existence of
   something using several languages
Polyglot Persistence
The continued or prolonged existence of
   something using several languages
              databases
“Polyglot Persistence, like
  polyglot programming, is all
    about choosing the right
persistence option for the task at
   hand.” - Scott Leberknight,
         October, 2008


     http://www.nearinfinity.com/blogs/scott_leberknight/polyglot_persistence.html
Why On Earth Would
You Want To Do This?
CAP Theorem



  http://en.wikipedia.org/wiki/CAP_theorem
http://blog.nahurst.com/visual-guide-to-nosql-systems
Compromise
Consistency and
 Data Integrity
       +
 Scalability and
   Flexibility
Support A Wide Range
     of Storage
   Requirements
Get The Job Done
Faster, With Better
     Quality
DB Doesn't Just Stand For
       Database
Don't Swim Upstream
Possible Use Cases
Use A NoSQL Database
    For A Particular
  Application Feature
Use A NoSQL Database
  For Speedy Batch
      Processing
Use A NoSQL Database
For Distributed Logging
Use A NoSQL Database
   For Large Tables
Use A RDBMS For
    Reporting
Sounds Great!
What's The Catch?
Difficult For Data In
Different Databases To
        Interact
You Now Have To
Decide Where To Store
        Data
Increased Application
  And Deployment
     Complexity
Additional
Administrative
Responsibilities
Training
Polyglot Persistence - Two Great Tastes That Taste Great Together
What Will This Do To
My Beautiful Code?
It's All About The Layers
class User < ActiveRecord::Base
end


class ContestEntry < CouchRest::ExtendedDocument
 property :entry_number
end
class User < ActiveRecord::Base
 def contest_entries
   ContestEntry.entries_for_user(self.id)
 end
end

class ContestEntry < CouchRest::ExtendedDocument
 property :entry_number
 property :user_id

 def self.entries_for_user(user_id)
  # Execute your view to fetch the contest entries
 end

 def user
  User.f nd_by_id(user_id)
       i
 end
end
Additional Options
    Available
So, Who Is Actually
    Doing This?
Polyglot Persistence - Two Great Tastes That Taste Great Together
●   Primary MySQL database with a backup
●   A few very large tables, containing 5M – 30M
    rows each, and growing quickly
●   Increasing query execution time
●   Some pages on the web app were timing out
●   Increasing database migration time
●   Rigid schema of the RDBMS was preventing
    some planned features from moving forward
●   Brought in a consultant to help us optimize our
    MySQL setup
●   Optimized slow queries
●   Added some indexes
●   Offloaded some work to the backup database
●   Considered the use of summary tables for
    statistics
+
●   Migrated old data from large tables to CouchDB
●   Using CouchDB views to aggregate summary
    data
●   Data is imported and views are updated nightly
●   Queries for statistics now very fast
●   Using Lucene (via couchdb-lucene) for full text
    searching
●   Taking full advantage of CouchDBs schema-
    less nature in several new application features
It's Not All Rainbows And Unicorns
●   CouchDB databases and views can be very
    large on disk
●   Some queries could not be substituted with
    CouchDB views
●   Indexing tens of millions of documents for full
    text search with Lucene takes weeks
●   Development takes longer, as the map/reduce
    model requires additional thought and planning
●   Changing/Upgrading views in production not
    straightforward
                  http://www.couch.io/migrating-to-couchdb
Polyglot Persistence - Two Great Tastes That Taste Great Together
http://twitter.com/about/opensource
Polyglot Persistence - Two Great Tastes That Taste Great Together
●   Vertically and horizontally partitioned MySQL
●   Several layers of aggressive caching, all
    application managed
●   Schema changes impossible, resulting in the
    use of bitfields and piggyback tables
●   Hardware intensive
●   Error prone
●   Hitting MySQL limits
●   Already eventually consistent
HBase



FlockDB
●   Migrating from MySQL to Cassandra as their
    main online data store
●   Hadoop/HBase used for people search feature
●   FlockDB used to manage the social graph
●   Hadoop for analytics
●   “As with all NoSQL systems, strengths in
    different situations” - Kevin Weil, Analytics
    Lead, Twitter
             http://www.slideshare.net/kevinweil/nosql-at-twitter-nosql-eu-2010
●   Increased availability
●   The ability to support new features
●   The ability to analyze their massive amount of
    data in a reasonable amount of time




            http://www.slideshare.net/kevinweil/nosql-at-twitter-nosql-eu-2010
Right Tool For The Job
Thanks!
john_p_wood@yahoo.com
      @johnpwood

Weitere ähnliche Inhalte

Was ist angesagt?

Couchbase training basic
Couchbase training basicCouchbase training basic
Couchbase training basicKnoldus Inc.
 
20171019 data migration (rk)
20171019 data migration (rk)20171019 data migration (rk)
20171019 data migration (rk)Ruud Kapteijn
 
QuerySurge Slide Deck for Big Data Testing Webinar
QuerySurge Slide Deck for Big Data Testing WebinarQuerySurge Slide Deck for Big Data Testing Webinar
QuerySurge Slide Deck for Big Data Testing WebinarRTTS
 
Data Lake Overview
Data Lake OverviewData Lake Overview
Data Lake OverviewJames Serra
 
How does Microsoft solve Big Data?
How does Microsoft solve Big Data?How does Microsoft solve Big Data?
How does Microsoft solve Big Data?James Serra
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureJames Serra
 
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Amazon Web Services
 
Accelerate and modernize your data pipelines
Accelerate and modernize your data pipelinesAccelerate and modernize your data pipelines
Accelerate and modernize your data pipelinesPaul Van Siclen
 
Big Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and ZeppelinBig Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and Zeppelinprajods
 
Snowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleSnowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleAdam Doyle
 
Microsoft Power BI
Microsoft Power BIMicrosoft Power BI
Microsoft Power BISushil kasar
 
A Roadmap to Data Migration Success
A Roadmap to Data Migration SuccessA Roadmap to Data Migration Success
A Roadmap to Data Migration SuccessFindWhitePapers
 
Lessons Learned - Monitoring the Data Pipeline at Hulu
Lessons Learned - Monitoring the Data Pipeline at HuluLessons Learned - Monitoring the Data Pipeline at Hulu
Lessons Learned - Monitoring the Data Pipeline at HuluDataWorks Summit
 
NOSQL Databases types and Uses
NOSQL Databases types and UsesNOSQL Databases types and Uses
NOSQL Databases types and UsesSuvradeep Rudra
 
Introducing Azure SQL Data Warehouse
Introducing Azure SQL Data WarehouseIntroducing Azure SQL Data Warehouse
Introducing Azure SQL Data WarehouseJames Serra
 
A 30 day plan to start ending your data struggle with Snowflake
A 30 day plan to start ending your data struggle with SnowflakeA 30 day plan to start ending your data struggle with Snowflake
A 30 day plan to start ending your data struggle with SnowflakeSnowflake Computing
 

Was ist angesagt? (20)

IBM - Introduction to Cloudant
IBM - Introduction to CloudantIBM - Introduction to Cloudant
IBM - Introduction to Cloudant
 
Couchbase training basic
Couchbase training basicCouchbase training basic
Couchbase training basic
 
20171019 data migration (rk)
20171019 data migration (rk)20171019 data migration (rk)
20171019 data migration (rk)
 
QuerySurge Slide Deck for Big Data Testing Webinar
QuerySurge Slide Deck for Big Data Testing WebinarQuerySurge Slide Deck for Big Data Testing Webinar
QuerySurge Slide Deck for Big Data Testing Webinar
 
Data Lake Overview
Data Lake OverviewData Lake Overview
Data Lake Overview
 
How does Microsoft solve Big Data?
How does Microsoft solve Big Data?How does Microsoft solve Big Data?
How does Microsoft solve Big Data?
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
 
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
 
Accelerate and modernize your data pipelines
Accelerate and modernize your data pipelinesAccelerate and modernize your data pipelines
Accelerate and modernize your data pipelines
 
Big Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and ZeppelinBig Data visualization with Apache Spark and Zeppelin
Big Data visualization with Apache Spark and Zeppelin
 
Snowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleSnowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at Scale
 
Microsoft Power BI
Microsoft Power BIMicrosoft Power BI
Microsoft Power BI
 
Oracle Database 12c : Multitenant
Oracle Database 12c : MultitenantOracle Database 12c : Multitenant
Oracle Database 12c : Multitenant
 
A Roadmap to Data Migration Success
A Roadmap to Data Migration SuccessA Roadmap to Data Migration Success
A Roadmap to Data Migration Success
 
Lessons Learned - Monitoring the Data Pipeline at Hulu
Lessons Learned - Monitoring the Data Pipeline at HuluLessons Learned - Monitoring the Data Pipeline at Hulu
Lessons Learned - Monitoring the Data Pipeline at Hulu
 
Introduction to Couchbase
Introduction to CouchbaseIntroduction to Couchbase
Introduction to Couchbase
 
NOSQL Databases types and Uses
NOSQL Databases types and UsesNOSQL Databases types and Uses
NOSQL Databases types and Uses
 
Introducing Azure SQL Data Warehouse
Introducing Azure SQL Data WarehouseIntroducing Azure SQL Data Warehouse
Introducing Azure SQL Data Warehouse
 
Snowflake Datawarehouse Architecturing
Snowflake Datawarehouse ArchitecturingSnowflake Datawarehouse Architecturing
Snowflake Datawarehouse Architecturing
 
A 30 day plan to start ending your data struggle with Snowflake
A 30 day plan to start ending your data struggle with SnowflakeA 30 day plan to start ending your data struggle with Snowflake
A 30 day plan to start ending your data struggle with Snowflake
 

Ähnlich wie Polyglot Persistence - Two Great Tastes That Taste Great Together

NoSQLDatabases
NoSQLDatabasesNoSQLDatabases
NoSQLDatabasesAdi Challa
 
Big data and polyglot solutions
Big data and polyglot solutionsBig data and polyglot solutions
Big data and polyglot solutionsKumaran Ramanujam
 
Introduction to NoSQL and MongoDB
Introduction to NoSQL and MongoDBIntroduction to NoSQL and MongoDB
Introduction to NoSQL and MongoDBAhmed Farag
 
Practical Design Patterns for Building Applications Resilient to Infrastructu...
Practical Design Patterns for Building Applications Resilient to Infrastructu...Practical Design Patterns for Building Applications Resilient to Infrastructu...
Practical Design Patterns for Building Applications Resilient to Infrastructu...MongoDB
 
Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?
Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?
Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?brianlangbecker
 
Big data & frameworks: no book for you anymore.
Big data & frameworks: no book for you anymore.Big data & frameworks: no book for you anymore.
Big data & frameworks: no book for you anymore.Roman Nikitchenko
 
Big data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymoreBig data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymoreStfalcon Meetups
 
Getting Started with Big Data in the Cloud
Getting Started with Big Data in the CloudGetting Started with Big Data in the Cloud
Getting Started with Big Data in the CloudRightScale
 
Hadoop Training Tutorial for Freshers
Hadoop Training Tutorial for FreshersHadoop Training Tutorial for Freshers
Hadoop Training Tutorial for Freshersrajkamaltibacademy
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQLbalwinders
 
Couch DB/PouchDB approach for hybrid mobile applications
Couch DB/PouchDB approach for hybrid mobile applicationsCouch DB/PouchDB approach for hybrid mobile applications
Couch DB/PouchDB approach for hybrid mobile applicationsIhor Malytskyi
 
Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Andrew Brust
 
Moving data to the cloud BY CESAR ROJAS from Pivotal
Moving data to the cloud BY CESAR ROJAS from PivotalMoving data to the cloud BY CESAR ROJAS from Pivotal
Moving data to the cloud BY CESAR ROJAS from PivotalVMware Tanzu Korea
 
Proud to be polyglot
Proud to be polyglotProud to be polyglot
Proud to be polyglotTugdual Grall
 
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ PanoraysQuick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ PanoraysDemi Ben-Ari
 
How To Tell if Your Business Needs NoSQL
How To Tell if Your Business Needs NoSQLHow To Tell if Your Business Needs NoSQL
How To Tell if Your Business Needs NoSQLDataStax
 
Big Data (NJ SQL Server User Group)
Big Data (NJ SQL Server User Group)Big Data (NJ SQL Server User Group)
Big Data (NJ SQL Server User Group)Don Demcsak
 

Ähnlich wie Polyglot Persistence - Two Great Tastes That Taste Great Together (20)

NoSQLDatabases
NoSQLDatabasesNoSQLDatabases
NoSQLDatabases
 
Big data and polyglot solutions
Big data and polyglot solutionsBig data and polyglot solutions
Big data and polyglot solutions
 
BigData Hadoop
BigData Hadoop BigData Hadoop
BigData Hadoop
 
Introduction to NoSQL and MongoDB
Introduction to NoSQL and MongoDBIntroduction to NoSQL and MongoDB
Introduction to NoSQL and MongoDB
 
Practical Design Patterns for Building Applications Resilient to Infrastructu...
Practical Design Patterns for Building Applications Resilient to Infrastructu...Practical Design Patterns for Building Applications Resilient to Infrastructu...
Practical Design Patterns for Building Applications Resilient to Infrastructu...
 
Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?
Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?
Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?
 
Big data & frameworks: no book for you anymore.
Big data & frameworks: no book for you anymore.Big data & frameworks: no book for you anymore.
Big data & frameworks: no book for you anymore.
 
Big data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymoreBig data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymore
 
Getting Started with Big Data in the Cloud
Getting Started with Big Data in the CloudGetting Started with Big Data in the Cloud
Getting Started with Big Data in the Cloud
 
Hadoop Training Tutorial for Freshers
Hadoop Training Tutorial for FreshersHadoop Training Tutorial for Freshers
Hadoop Training Tutorial for Freshers
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
 
Couch DB/PouchDB approach for hybrid mobile applications
Couch DB/PouchDB approach for hybrid mobile applicationsCouch DB/PouchDB approach for hybrid mobile applications
Couch DB/PouchDB approach for hybrid mobile applications
 
Architecting Your First Big Data Implementation
Architecting Your First Big Data ImplementationArchitecting Your First Big Data Implementation
Architecting Your First Big Data Implementation
 
Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Big Data Strategy for the Relational World
Big Data Strategy for the Relational World
 
Moving data to the cloud BY CESAR ROJAS from Pivotal
Moving data to the cloud BY CESAR ROJAS from PivotalMoving data to the cloud BY CESAR ROJAS from Pivotal
Moving data to the cloud BY CESAR ROJAS from Pivotal
 
Proud to be polyglot
Proud to be polyglotProud to be polyglot
Proud to be polyglot
 
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ PanoraysQuick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
 
How To Tell if Your Business Needs NoSQL
How To Tell if Your Business Needs NoSQLHow To Tell if Your Business Needs NoSQL
How To Tell if Your Business Needs NoSQL
 
Big Data (NJ SQL Server User Group)
Big Data (NJ SQL Server User Group)Big Data (NJ SQL Server User Group)
Big Data (NJ SQL Server User Group)
 
Rails data migrations
Rails data migrationsRails data migrations
Rails data migrations
 

Kürzlich hochgeladen

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 

Kürzlich hochgeladen (20)

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 

Polyglot Persistence - Two Great Tastes That Taste Great Together

  • 1. Polyglot Persistence Two Great Tastes That Taste Great Together! John Wood john_p_wood@yahoo.com @johnpwood
  • 2. About Me ● Software Developer at Interactive Mediums ● Primarily work on a web application that allows our customers to engage and interact with their customers ● Writing code for about 15 years ● Tinkering with NoSQL for about 1.5 years ● Have a NoSQL solution that has been running in production for a year
  • 3. You Now Have A Choice
  • 4. You Now Have A Choice
  • 5. You Now Have A Choice
  • 6. You Now Have A Choice
  • 7. You Now Have A Choice
  • 8. You Now Have A Choice
  • 9. You Now Have A Choice
  • 10. You Now Have A Choice
  • 11. You Now Have A Choice
  • 12. You Now Have A Choice
  • 13. The RDBMS Is No Longer The Default Choice
  • 14. The RDBMS Is No Longer The Default Choice ● Can be very difficult to scale horizontally ● Schemas can be difficult to maintain and migrate ● For some applications, the data integrity features of the RDBMS are an unnecessary overhead ● Data constraints and JOINs can be expensive at runtime
  • 15. NoSQL Databases Have Stepped Up To Address These Issues
  • 16. NoSQL Databases Have Stepped Up To Address These Issues ● Schema-less ● Little to no data integrity enforcement ● Self-contained data ● Eventually consistent ● Easy to scale horizontally to add processing power and storage
  • 17. But The RDBMS Is Far From Dead
  • 18. But The RDBMS Is Far From Dead ● Incredibly mature, and battle tested ● Immediate and constant consistency ● Integrity of data is enforced ● Efficient use of storage space if data normalized properly ● Supported by everyone and everything (tools, frameworks, libraries, etc) ● Incredibly flexible and powerful query language ● Help is plentiful and easy to find
  • 21. You Don't Have to Choose
  • 22. “You've got your chocolate in my peanut butter!”
  • 24. pol●y●glot - Adjective Knowing or using several languages
  • 25. pol●y●glot - Adjective Knowing or using several languages per●sist●ence - Noun The continued or prolonged existence of something
  • 26. Polyglot Persistence The continued or prolonged existence of something using several languages
  • 27. Polyglot Persistence The continued or prolonged existence of something using several languages databases
  • 28. “Polyglot Persistence, like polyglot programming, is all about choosing the right persistence option for the task at hand.” - Scott Leberknight, October, 2008 http://www.nearinfinity.com/blogs/scott_leberknight/polyglot_persistence.html
  • 29. Why On Earth Would You Want To Do This?
  • 30. CAP Theorem http://en.wikipedia.org/wiki/CAP_theorem
  • 33. Consistency and Data Integrity + Scalability and Flexibility
  • 34. Support A Wide Range of Storage Requirements
  • 35. Get The Job Done Faster, With Better Quality
  • 36. DB Doesn't Just Stand For Database
  • 39. Use A NoSQL Database For A Particular Application Feature
  • 40. Use A NoSQL Database For Speedy Batch Processing
  • 41. Use A NoSQL Database For Distributed Logging
  • 42. Use A NoSQL Database For Large Tables
  • 43. Use A RDBMS For Reporting
  • 45. Difficult For Data In Different Databases To Interact
  • 46. You Now Have To Decide Where To Store Data
  • 47. Increased Application And Deployment Complexity
  • 51. What Will This Do To My Beautiful Code?
  • 52. It's All About The Layers
  • 53. class User < ActiveRecord::Base end class ContestEntry < CouchRest::ExtendedDocument property :entry_number end
  • 54. class User < ActiveRecord::Base def contest_entries ContestEntry.entries_for_user(self.id) end end class ContestEntry < CouchRest::ExtendedDocument property :entry_number property :user_id def self.entries_for_user(user_id) # Execute your view to fetch the contest entries end def user User.f nd_by_id(user_id) i end end
  • 55. Additional Options Available
  • 56. So, Who Is Actually Doing This?
  • 58. Primary MySQL database with a backup ● A few very large tables, containing 5M – 30M rows each, and growing quickly ● Increasing query execution time ● Some pages on the web app were timing out ● Increasing database migration time ● Rigid schema of the RDBMS was preventing some planned features from moving forward
  • 59. Brought in a consultant to help us optimize our MySQL setup ● Optimized slow queries ● Added some indexes ● Offloaded some work to the backup database ● Considered the use of summary tables for statistics
  • 60. +
  • 61. Migrated old data from large tables to CouchDB ● Using CouchDB views to aggregate summary data ● Data is imported and views are updated nightly ● Queries for statistics now very fast ● Using Lucene (via couchdb-lucene) for full text searching ● Taking full advantage of CouchDBs schema- less nature in several new application features
  • 62. It's Not All Rainbows And Unicorns
  • 63. CouchDB databases and views can be very large on disk ● Some queries could not be substituted with CouchDB views ● Indexing tens of millions of documents for full text search with Lucene takes weeks ● Development takes longer, as the map/reduce model requires additional thought and planning ● Changing/Upgrading views in production not straightforward http://www.couch.io/migrating-to-couchdb
  • 67. Vertically and horizontally partitioned MySQL ● Several layers of aggressive caching, all application managed ● Schema changes impossible, resulting in the use of bitfields and piggyback tables ● Hardware intensive ● Error prone ● Hitting MySQL limits ● Already eventually consistent
  • 69. Migrating from MySQL to Cassandra as their main online data store ● Hadoop/HBase used for people search feature ● FlockDB used to manage the social graph ● Hadoop for analytics ● “As with all NoSQL systems, strengths in different situations” - Kevin Weil, Analytics Lead, Twitter http://www.slideshare.net/kevinweil/nosql-at-twitter-nosql-eu-2010
  • 70. Increased availability ● The ability to support new features ● The ability to analyze their massive amount of data in a reasonable amount of time http://www.slideshare.net/kevinweil/nosql-at-twitter-nosql-eu-2010
  • 71. Right Tool For The Job