SlideShare a Scribd company logo
1 of 35
Data Modeling for NoSQL
Tony Tam
@fehguy
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/data-modeling-mongodb
Presented at QCon San Francisco
www.qconsf.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Data Modeling?!
Smart
Modeling
makes NoSQL
Why Modeling Matters!
• NoSQL => no joins!
• What replaces joins?!
•  Hierarchy!
•  Duplication of data!
•  Different models for querying, indexing!
• Your optimal data model is (probably) very
different than with relational!
•  Simpler!
•  More like you develop!
Stop Thinking Like This!!
endless layers
of abstraction	

(and misery)
Hierarchy before NoSQL!
• Simple User Model!
Hierarchy before NoSQL!
• Tuned Queries!
•  Write some brittle SQL:!
•  “select user.id, … inner join settings on …!
•  Pick out the fields and construct object hierarchy
(this gets nasty, fast)!
•  (outer joins for optional values?)!
• Object fetching!
•  Queries follow object graph, PK/FK!
•  5 queries to fetch object in this example!
Hierarchy before NoSQL!
Hierarchy with NoSQL!
• JSON structure mapped to objects!
•  Fetch json from MongoDB**!
•  Unmarshall into objects/tuples!
•  Use it!
Using JSON4S
Hierarchy with NoSQL!
Focus on your
Software, not
DB layer!
Hierarchy with NoSQL!
• Write operations!
•  Atomic upsert (create, update or fail)!
!
•  Saves all levels of object atomically!
•  Reduces need for transactions!
Hierarchy with NoSQL!
• Write operations!
•  Atomic upsert (create, update or fail)!
!
•  Saves all levels of object atomically!
•  Reduces need for transactions!
All or
Convenienc
not magic
Unique Identiers in your Data!
• Relational design => PK/FK!
•  Often not “meaningful” identifiers for data!
• User Data Model!
Unique Identiers in your Data!
• Relational design => PK/FK!
•  Often not “meaningful” identifiers for data!
• User Data Model!
Unique by
username
Unique Identiers in your Data!
• Words! Ensured to
be constant
Data Duplication !
• Without Joins, what about SQL lookup
tables?!
•  Duplication of data in NoSQL is required!
• Trade storage for speed!
Data Duplication !
• Without Joins, what about SQL lookup
tables?!
•  Duplication of data in NoSQL is required!
• Trade storage for speed!
…Can move
logic to app
Data Duplication!
• Many fields don’t change, ever!
• But… many do!
•  New decisions for the developer!!
•  Often background updates!
Data Duplication!
• Many fields don’t change, ever!
• But… many do!
•  New decisions for the developer!!
•  Often background updates!
How often
does this
change?
Data Duplication!
Reaching into Objects!
• Incredible feature of MongoDB!
•  Dot syntax safely** traverses the object graph!
Inner Indexes!
• Convenience at a cost!
•  No index => table scan!
•  No value? => table scan!
•  No child value? => table scan!
• Table scan with big collection?!
• Can’t index everything!!
96GB of
Indexes?
Inner Indexes!
• This will should drive your Data Model!
• Sparse Data test!
Even with only
2000 non-empty
values!
Adding & Modifying!
• Append in mongo is blazing fast!
•  “tail” of data is always in memory!
•  Pre-allocated data files!
• Main expense is “index maintenance”!
•  Some marshalling/unmarshalling cost**!
• Modifying? Object growth!
•  Pre-allocation of space built in collection design!
Adding & Modifying!
• Each object has allocated space!
•  Exceed that space, need to relocate object!
•  Leaves “hole” in collection!
• Large increases to documents hurts your
overall performance!
• Your data model should strive for equally-
sized objects as much as possible!
Retrieval!
• Many same rules apply as relational!
• Indexes !
•  complex/inner or not!
•  Indexes in RAM? Yes!
•  Cardinality matters!
• New(ish) considerations!
•  Complex hierarchy not free!
•  Marshalling ó unmarshalling!
Marshalling & Unmarshalling!
Object
complexity
Marshalling & Unmarshalling!
• All you can eat from your Data Model?!
• Techniques have tremendous impact!
•  Development ease until it matters!
•  50% speed bump with manual mapping!
Only demand
what you can
consume!
Making the most of _id!
• Indexes matter!
• Tailor your _id to be meaningful by access
pattern!
•  It’s your first defense when auto-sharding!
• Date-driven data?!
•  Monotonically _id value!
•  Ensures recent data is “hot”!
Making the most of _id!
• Other time-based data techniques!
• Flexibility in querying!
Making the most of _id!
• Other time-based data techniques!
• Flexibility in querying!
Case-
sensitive
REGEX is
Making the most of _id!
• Hot indexes are happy indexes!
•  Access should strive for right bias!
• Random access with large indexes hit disk
17	

15
Your Data Model!
• NoSQL gets you started faster!
• Many relational pain points are gone!
• New considerations (easier?)!
• Migration should be real effort!
• Designed by access patterns over object
structure!
• Don’t prematurely optimize, but know
where the knobs are!
More Reading!
• http://tech.wordnik.com!
• http://github.com/wordnik/wordnik-oss!
• http://developer.wordnik.com!
• http://slideshare.net/fehguy!

More Related Content

Viewers also liked

6 Data Modeling for NoSQL 2/2
6 Data Modeling for NoSQL 2/26 Data Modeling for NoSQL 2/2
6 Data Modeling for NoSQL 2/2
Fabio Fumarola
 
Modeling Data in MongoDB
Modeling Data in MongoDBModeling Data in MongoDB
Modeling Data in MongoDB
lehresman
 
Common MongoDB Use Cases
Common MongoDB Use Cases Common MongoDB Use Cases
Common MongoDB Use Cases
MongoDB
 
5 Data Modeling for NoSQL 1/2
5 Data Modeling for NoSQL 1/25 Data Modeling for NoSQL 1/2
5 Data Modeling for NoSQL 1/2
Fabio Fumarola
 

Viewers also liked (8)

6 Data Modeling for NoSQL 2/2
6 Data Modeling for NoSQL 2/26 Data Modeling for NoSQL 2/2
6 Data Modeling for NoSQL 2/2
 
Data Modeling for NoSQL
Data Modeling for NoSQLData Modeling for NoSQL
Data Modeling for NoSQL
 
Modeling Data in MongoDB
Modeling Data in MongoDBModeling Data in MongoDB
Modeling Data in MongoDB
 
11. From Hadoop to Spark 2/2
11. From Hadoop to Spark 2/211. From Hadoop to Spark 2/2
11. From Hadoop to Spark 2/2
 
Common MongoDB Use Cases
Common MongoDB Use Cases Common MongoDB Use Cases
Common MongoDB Use Cases
 
5 Data Modeling for NoSQL 1/2
5 Data Modeling for NoSQL 1/25 Data Modeling for NoSQL 1/2
5 Data Modeling for NoSQL 1/2
 
MongoDB Schema Design
MongoDB Schema DesignMongoDB Schema Design
MongoDB Schema Design
 
MongoDB Schema Design: Four Real-World Examples
MongoDB Schema Design: Four Real-World ExamplesMongoDB Schema Design: Four Real-World Examples
MongoDB Schema Design: Four Real-World Examples
 

More from C4Media

More from C4Media (20)

Streaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoStreaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
 
Next Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy MobileNext Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy Mobile
 
Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020
 
Understand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java ApplicationsUnderstand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java Applications
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No Keeper
 
High Performing Teams Act Like Owners
High Performing Teams Act Like OwnersHigh Performing Teams Act Like Owners
High Performing Teams Act Like Owners
 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaDoes Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
 
Service Meshes- The Ultimate Guide
Service Meshes- The Ultimate GuideService Meshes- The Ultimate Guide
Service Meshes- The Ultimate Guide
 
Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CD
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine Learning
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at Speed
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep Systems
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.js
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly Compiler
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix Scale
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's Edge
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home Everywhere
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing For
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data Engineering
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
Christopher Logan Kennedy
 

Recently uploaded (20)

Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 

Data Modeling for NoSQL

  • 1. Data Modeling for NoSQL Tony Tam @fehguy
  • 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /data-modeling-mongodb
  • 3. Presented at QCon San Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  • 5. Why Modeling Matters! • NoSQL => no joins! • What replaces joins?! •  Hierarchy! •  Duplication of data! •  Different models for querying, indexing! • Your optimal data model is (probably) very different than with relational! •  Simpler! •  More like you develop!
  • 6. Stop Thinking Like This!! endless layers of abstraction (and misery)
  • 8. Hierarchy before NoSQL! • Tuned Queries! •  Write some brittle SQL:! •  “select user.id, … inner join settings on …! •  Pick out the elds and construct object hierarchy (this gets nasty, fast)! •  (outer joins for optional values?)! • Object fetching! •  Queries follow object graph, PK/FK! •  5 queries to fetch object in this example!
  • 10. Hierarchy with NoSQL! • JSON structure mapped to objects! •  Fetch json from MongoDB**! •  Unmarshall into objects/tuples! •  Use it! Using JSON4S
  • 11. Hierarchy with NoSQL! Focus on your Software, not DB layer!
  • 12. Hierarchy with NoSQL! • Write operations! •  Atomic upsert (create, update or fail)! ! •  Saves all levels of object atomically! •  Reduces need for transactions!
  • 13. Hierarchy with NoSQL! • Write operations! •  Atomic upsert (create, update or fail)! ! •  Saves all levels of object atomically! •  Reduces need for transactions! All or Convenienc not magic
  • 14. Unique Identiers in your Data! • Relational design => PK/FK! •  Often not “meaningful” identiers for data! • User Data Model!
  • 15. Unique Identiers in your Data! • Relational design => PK/FK! •  Often not “meaningful” identiers for data! • User Data Model! Unique by username
  • 16. Unique Identiers in your Data! • Words! Ensured to be constant
  • 17. Data Duplication ! • Without Joins, what about SQL lookup tables?! •  Duplication of data in NoSQL is required! • Trade storage for speed!
  • 18. Data Duplication ! • Without Joins, what about SQL lookup tables?! •  Duplication of data in NoSQL is required! • Trade storage for speed! …Can move logic to app
  • 19. Data Duplication! • Many elds don’t change, ever! • But… many do! •  New decisions for the developer!! •  Often background updates!
  • 20. Data Duplication! • Many elds don’t change, ever! • But… many do! •  New decisions for the developer!! •  Often background updates! How often does this change?
  • 22. Reaching into Objects! • Incredible feature of MongoDB! •  Dot syntax safely** traverses the object graph!
  • 23. Inner Indexes! • Convenience at a cost! •  No index => table scan! •  No value? => table scan! •  No child value? => table scan! • Table scan with big collection?! • Can’t index everything!! 96GB of Indexes?
  • 24. Inner Indexes! • This will should drive your Data Model! • Sparse Data test! Even with only 2000 non-empty values!
  • 25. Adding & Modifying! • Append in mongo is blazing fast! •  “tail” of data is always in memory! •  Pre-allocated data les! • Main expense is “index maintenance”! •  Some marshalling/unmarshalling cost**! • Modifying? Object growth! •  Pre-allocation of space built in collection design!
  • 26. Adding & Modifying! • Each object has allocated space! •  Exceed that space, need to relocate object! •  Leaves “hole” in collection! • Large increases to documents hurts your overall performance! • Your data model should strive for equally- sized objects as much as possible!
  • 27. Retrieval! • Many same rules apply as relational! • Indexes ! •  complex/inner or not! •  Indexes in RAM? Yes! •  Cardinality matters! • New(ish) considerations! •  Complex hierarchy not free! •  Marshalling ó unmarshalling!
  • 29. Marshalling & Unmarshalling! • All you can eat from your Data Model?! • Techniques have tremendous impact! •  Development ease until it matters! •  50% speed bump with manual mapping! Only demand what you can consume!
  • 30. Making the most of _id! • Indexes matter! • Tailor your _id to be meaningful by access pattern! •  It’s your rst defense when auto-sharding! • Date-driven data?! •  Monotonically _id value! •  Ensures recent data is “hot”!
  • 31. Making the most of _id! • Other time-based data techniques! • Flexibility in querying!
  • 32. Making the most of _id! • Other time-based data techniques! • Flexibility in querying! Case- sensitive REGEX is
  • 33. Making the most of _id! • Hot indexes are happy indexes! •  Access should strive for right bias! • Random access with large indexes hit disk 17 15
  • 34. Your Data Model! • NoSQL gets you started faster! • Many relational pain points are gone! • New considerations (easier?)! • Migration should be real effort! • Designed by access patterns over object structure! • Don’t prematurely optimize, but know where the knobs are!