Arthur Viegers, Senior Solutions Architect, MongoDB.
The value of the fast growing class of NoSQL databases is the ability to handle high velocity and volumes of data while enabling greater agility with dynamic schemas. MongoDB gives you those benefits while also providing a rich querying capability and a document model for developer productivity. Arthur Viegers outlines the reasons for MongoDB's popularity in IoT applications and how you can leverage the core concepts of NoSQL to build robust and highly scalable IoT applications.
8. Time series schema design goal
*
• Store event data
• Support Analytical Queries
• Find best compromise of:
- Memory utilization
- Write performance
- Read/Analytical Query Performance
• Accomplish with realistic amount of hardware
9. Modelling time series data
*
• Document per event
• Document per minute (average)
• Document per minute (second)
• Document per hour
11. Document per minute (average)
{
*
deviceId: "Test123",
timestamp: ISODate("2014-07-03T22:07:00.000Z"),
temperature_num: 18,
temperature_sum: 357
}
• Pre-aggregate to compute average per minute
more easily
• Update-driven workload
• Resolution at the minute level
12. Document per minute (by second)
{
*
deviceId: "Test123",
timestamp: ISODate("2014-07-03T22:07:00.000Z"),
temperature: { 0: 18, 1: 18, …, 58: 21, 59: 21 }
}
• Store per-second data at the minute level
• Update-driven workload
• Pre-allocate structure to avoid document moves
13. Document per hour (by second)
{
*
deviceId: "Test123",
timestamp: ISODate("2014-07-03T22:00:00.000Z"),
temperature: { 0: 18, 1: 18, …, 3598: 20, 3599: 20 }
}
• Store per-second data at the hourly level
• Update-driven workload
• Pre-allocate structure to avoid document moves
• Updating last second requires 3599 steps
14. Document per hour (by second)
{
*
deviceId: "Test123",
timestamp: ISODate("2014-07-03T22:00:00.000Z"),
temperature: {
0: { 0: 18, …, 59: 18 },
…,
59: { 0: 21, …, 59: 20 }
}
}
• Store per-second data at the hourly level with nesting
• Update-driven workload
• Pre-allocate structure to avoid document moves
• Updating last second requires 59 + 59 steps
24. Shard Key Considerations
*
• Cardinality
• Write distribution
• Query isolation
• Reliability
• Index locality
25. Shard Key Selection Rexroth NEXO
*
Cardinality
Write
Distribution
Query Isolation Reliability
Index
Locality
_id Doc level One shard Scatter/gather
All users
affected
Good
hash(_id) Hash level All Shards Scatter/gather
All users
affected
Poor
assetId Many docs All Shards Targeted
Some assets
affected
Good
assetId, hour Doc level All Shards Targeted
Some assets
affected
Good
26. Scaling Data - Summarized
*
• MongoDB scales horizontally (sharding)
• Each shard is an independent database, and
collectively, the shards make up a single logical
database
• MMS makes it easy and reliable to run
MongoDB at scale
• Sharding requires minimal effort from the
application code: same interface as single
mongod
28. Why is MongoDB a good fit for IoT?
*
• IoT processes are real-time
• Relational technologies can simply not compete on cost,
performance, scalability, and manageability
• IoT data can come in any format, structured or
unstructured, ranging from text and numbers to
audio, picture and video
• Time series data is a natural fit
• IoT applications often require geographically
distributed systems