MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
•Als PPTX, PDF herunterladen•
5 gefällt mir•2,191 views
The Sage Data Cloud enables next-generation cloud and mobile services via a Hybrid Cloud and Polyglot Persistence Architecture. Come learn how MongoDB and other cloud data stores make this a reality, and get an insight into our learnings and operations.
![What is this about?
db.Talk.find({
learnings: {
$in: [
"Polyglot Persistence & Architecture",
"How we use MongoDB",
"Tools & Ops"
]
}
});
2](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (12)
Ähnlich wie MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
Ähnlich wie MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture (20)
Mehr von MongoDB
Mehr von MongoDB (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
- 1. MongoDB in the middle of a Polyglot Persistence Architecture Sigfrido Narváez Principal Software Architect Sage North America MongoDB World 2015 June 2015 NY
- 2. What is this about? db.Talk.find({ learnings: { $in: [ "Polyglot Persistence & Architecture", "How we use MongoDB", "Tools & Ops" ] } }); 2
- 3. System of Record ERP, Accounting, HR & Payroll ERP’s On-premise SMB Sage Data Cloud Connector Customers Sales Product Catalog Payroll 3 Domains
- 4. 4
- 5. Data Taxonomy Types Reference Transactional Historical Computed Metadata Settings App-specific data Files & Logs Global / Lookup Behavior Origin: ERP / Cloud Querying Fixed Unstructured Growth Throughput Searchable Retention Binary
- 8. 8 Before
- 9. 9 … now!
- 11. Data Taxonomy Types Reference Transactional Historical Computed Metadata Settings App-specific data Files & Logs Global / Lookup Behavior Origin: ERP / Cloud Querying Fixed Schema Unstructured Growth Throughput Searchable Retention Binary
- 12. Abstractions
- 13. Sage ERP’s On-premise Domains as Micro-Services Connector 13 Customers Sales Products Catalog Payroll REST API Resources CRUD & Operations Business Entities Rules Data Layer Mappings Migrations ERP Plug-in Contracts Mappings Connector Contracts Background Data Sync Background Data Sync
- 15. 6/5/2015 15 Apps & FaaS built on top of Domains Each Governs Entities, Rules & Persistence Organized around Business Capabilities New Apps can re-use, extend or add New Technologies can be added or replace existing
- 16. 16 MongoDB • Use Cases • Aggregation Framework • Lessons Learned
- 17. MongoDB Use Cases • Historical Data • Write bursts • Mongo handles easily! • KPI’s via Aggregation Framework
- 18. 18 REST API Business Data Layer KPI Historical Azure Worker Aggregation Pipeline Pre-Calculated On-The-Fly Aggregation Framework { { "$match", { .. } }, { "$sort", { .. } }, { "$limit", 50} { "$unwind", { .. } }, { "$match", { .. } }, { "$limit", 50}, { "$project", { .. } } }
- 19. Dynamic schema • Settings • Tenant, User, App, Features, etc. • Problematic in Relational model • Outgrows K/V model • App-specific Data • Extensions to Core Entities • i.e. a SalesPerson is an extension of a User • Personalization
- 20. SQL MongoDB { "_id" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066SalesSalesTeamMember", "CreatedOn" : ISODate("2014-09-15T17:12:48.917Z"), "UpdatedOn" : ISODate("2014-09-15T17:12:48.917Z"), "CreatedBy" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066", "UpdatedBy" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066", "TenantId" : 3, "EntityStatus" : 0, "SettingKey" : "SalesTeamMember", "AppId" : 0, "ExtendedEntityId" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066", "SalespersonIds" : [new BinData(3, "mcseUhOfQUmPK6OnAKhQfA==")], "Title" : "TestTitle", "IsSalesMgr" : false, "IsSalesRep" : false, "SalesRepMaxDiscPct" : 0.0 } { "_id" : "2e866412-dbda-47ad-8767-a3a700a78e21ServiceServicesUser", "CreatedOn" : ISODate("2014-09-15T17:10:03.04Z"), "UpdatedOn" : ISODate("2014-09-15T17:10:03.04Z"), "CreatedBy" : "00000000-0000-0000-0000-000000000000", "UpdatedBy" : "00000000-0000-0000-0000-000000000000", "TenantId" : 3, "EntityStatus" : 0, "SettingKey" : "ServicesUser", "AppId" : 1, "ExtendedEntityId" : "2e866412-dbda-47ad-8767-a3a700a78e21", "Notes" : "Test Notes", "Role" : 1, "RoleAsString" : "Administrator", "LunchStartTime" : "12:00:00", "LunchEndTime" : "13:00:00", "StartTimeMon" : "08:00:00", "StartTimeTue" : "08:00:00", "StartTimeWed" : "08:00:00", "StartTimeThu" : "08:00:00", "StartTimeFri" : "08:00:00", "StartTimeSat" : "00:00:00", "StartTimeSun" : "00:00:00", "EndTimeMon" : "17:00:00", "EndTimeTue" : "17:00:00", "EndTimeWed" : "17:00:00", "EndTimeThu" : "17:00:00", "EndTimeFri" : "17:00:00", "EndTimeSat" : "00:00:00", "EndTimeSun" : "00:00:00" } Extensions @codinghorror
- 21. Lessons Learned Indexes • Adjust Indexes as needed • Background • Not on unique indexes! • Can’t add new background indexes and delete old indexes Migrations • Added after launch • DevOps “book keeping”, i.e. Indexes • https://github.com/phoenixwebgroup/DotNetMongoMigrations
- 22. Lessons Learned Exceed Namespace • 16mb limit / ~24K namespaces • Each collection & index is a namespace • Aggregation output to temp collections Aggregations • Ensure $match & $sort uses indexes • 100mb limit • Increase with allowDiskUse: true • Monitor for increase on Non-Mapped Virtual Memory
- 23. Lessons Learned • Aggregation $match – ensure uses indexes!
- 24. 6/5/2015 24 Operations • Azure • MMS • APM • CI/CD • Log Mining
- 25. Production DevOps - From code to alert paths! 25 Dev CI/CD Log Mining WAD, IIS Custom APM Azure Role Metrics DB Telemetry Deploy & DB Migrations code warm cold
- 26. Mongo on Azure Partnered with MongoLab • Dev & Production clusters • Slow queries & recommended indexes! (Dex) • Profiler & Stream logs • Telemetry: Real-Time Monitoring / link to MMS • http://docs.mongolab.com/monitoring On your own • Linux VM - http://docs.mongodb.org/ecosystem/tutorial/install-mongodb-on-linux-in- azure/ • Azure Worker - http://docs.mongodb.org/ecosystem/tutorial/deploy-mongodb- worker-roles-in-azure/
- 27. Mongo Monitoring & Performance (MMS) • Data, Index, Storage • Page Faults • Memory & CPU • Top 5 Alerts • Host Recovering, Repl Lag, Connections, Lock%, Replica • http://www.mongodb.com/blog/post/five-mms-monitoring-alerts-keep-your- mongodb-deployment-track • Top 5 Graphs • Lock %, Repl Lag, Journal Writes, Page faults, Non-Mapped Virtual Memory • http://blog.mongodb.org/post/62152249344/the-top-5-metrics-to-watch-in- mongodb
- 28. Non-mapped Virtual Memory Increase
- 29. Application Performance Monitoring MongoLab plug-in • Operations • Queues • Lock % • Page Faults • Non-mapped Virtual Memory • Memory • Replication Operations • Replication Lag • Oplog Window SQL • Query drill-down • Index profile
- 30. Continuous Integration / Delivery • Builds • Visual Studio aware • C# Code Analysis & Coverage • NUnit test execution • NuGet • Azure Deployments • Azure plugin • Runbook Azure Automation • Database Migrations • SQL • Mongo
- 31. Log Mining • Open Access (Engineers, DevOps, Customer Support) • Dump WAD, IIS & Error logs • Alerts & Dashboards • Verify issues have ceased (post deployment)
- 32. Alert Paths • Hot • Less than 10s • “Wake someone up” • Heartbeat (SCOM/NOC) • Warm • Less than 5m – DevOps engineer • NewRelic, MMS, Azure Portal • Slow Query, Slow Response, Repl Lag, Lock%, ApDex < 0.9, CPU% • Cold • Longer than 5m - Production Support / Engineering • SumoLogic – Error showed up in logs X times • Forensic purposes – saved queries
- 33. 33 Summary • Use the best of what is out there! • Divide for flexibility • MongoDB is amazing: multi-faceted • Enable DevOps with awesome tools! • Change is constant – Partner up!
- 34. THANK YOU! Sigfrido “Sig” Narváez • sigfrido@gmail.com | @SigNarvaez • Principal Software Architect, Sage • Orange County MUG Organizer - ~3yrs • 2015 MongoDB Master Sage • Solutions for ERP, CRM, HR, Payroll, Payments, Assets, Construction, and Real Estate. • Headquartered in UK, offices in North America, mainland Europe, South Africa, Australia, Asia and Brazil. • 13K Employees. 6M Customers. 23 Countries. • LSE:SGE
Hinweis der Redaktion
- NoSQL Joke
- Talk & Conversation Questions through out, no Q&A at end
- Large install base Every ERP different Standardize in the cloud (apps & schema) – not repeat Sync up from ERP to Cloud – Transactions down to ERP Apps share common data – user experience & value/investment carries over Many apps & services, and also cloud-to-cloud integrations & future ecosystem 1st Version – ALL SQL & Monolithic Mention OTHER SAGE CLOUDS AND ISV/PARTNER What were the challenges & why cloud?
- 1st attempt all sql – minimal nosql (k/v) Monolothic inter-related schema Performance issues Eventually would hit db size limits Each app requires core data + own data Generate new data from apps & services (images, etc) Share between apps Grow & grow, some purged Different sources and so on
- Identified what we were up against first Solve later – find patterns & tools Reference from ERP Global Lookup Historical Data will grow, maybe unstructured Not a table, but a matrix
- Inspired by Martin Fowler We have entered an area where we have the right tool for the right job Not a single hammer for all
- Inspired by Martin Fowler We have entered an area where we have the right tool for the right job, not a single hammer for all
- We can’t make this on a classic layered cake Birthday Parties of Before This is what we had built the first time around – 3 layer cake with monolithic schema
- We need a new type of cake!! … one that… is not a cake!! Every layer in the arch is made up of smaller more autonomous pieces That is the birthday cake of today!
- SQL rich, expensive and slow and size caps WATS is poor, cheap and fast Needed something in between for throughput and with no limits Bridge the gap, but not needed the big Hadoop, or big search datastore, etc
- Guideline for us, not cast in stone Highlights of sql, mongo, wats
- Reduce dependency on data technology Still “punch-a-hole” to get the benefit of each, but keep some level of base behavior Integrated Tenant Data Isolation & Security Keep logic abstracted from persistence Wrap database drivers or context classes REST calls fan-out into multiple data-stores, return a single payload Data Ownership – ERP or Cloud? Base Data Models determine Owner & DataStore – Restricted logic based owner
- Architecture builds domains, domains build the sdc Domains are the foundation for the data-cloud and for apps Each domain is a service with a full stack including ERP counter part Calls from Mobile Apps can range from a single domain, single datastore Or fan-out into multiple domains, each into multiple data stores Biz Logic separated from persistence logic
- Acrobatics and Abs joke
- 2 Years worth of history – various KPI’s (MTD YTD, frequently purchased) Inventory Category Tree Settings are flags & sometimes complex data that did not fit a K/V model and didn’t want to do SQL tables for
- Generate millions of insight documents from raw historical data Started with nightly batch Realized we could do on-the-fly Value of Aggregation in Real-Time – Less data, disk, memory, $
- 2 Years worth of history – various KPI’s (MTD YTD, frequently purchased) Inventory Category Tree Settings are flags & sometimes complex data that did not fit a K/V model and didn’t want to do SQL tables for
- Allow Apps to “extend” SDC entities with their own data fields Support settings and configurations at different levels App, Tenant, User, etc. Leverage MongoDB’s flexible schema
- Also mention aggregation framework and temp collections – run out of namespace (16mb limit)
- Also mention aggregation framework and temp collections – run out of namespace (16mb limit)
- Joke about time-scale – pegged CPU was I the order of several days! Fixed on Feb 14!! Happy Valentines! Thanks MongoLab
- Hot Less than 10s “Wake someone up” Heartbeat (SCOM/NOC) Warm Less than 5m – DevOps engineer NewRelic, MMS, Azure Portal Slow Query, Slow Response, Repl Lag, Lock%, ApDex < 0.9, CPU% Cold Longer than 5m - Production Support / Engineering SumoLogic – Error showed up in logs X times Forensic purposes – saved queries
- Spawn clusters in minutes - Scale up & down Engineers can turn on profiler anytime and stream logs Database compaction too
- By using MMS we found the issue on Feb 14 Also run-away non-mapped vm 0 Replication lag helps us determine when to bounce clusters Non-mapped virtual memory issue
- Story about dev
- Story about how we use sumo to find issues Correct & monitor cease Performance too Turn on verbose logging
- Story about how we use sumo to find issues Correct & monitor cease Performance too Turn on verbose logging
- Polyglot Hammer – everything looks like a nail SQL eons ago – people moved out of flat files k/v’s into sql We now see the return to older paradigms Architecture: Polyglot & MicroServices - Extensible & flexible: Extend our data cloud quickly Upgrade independently Scale where needed Mongo: Key piece of the puzzle missing in Azure to fulfill vision, we couldn’t have done it w/o! Very multi-faceted database that can be used for many data behaviors Bridges the gap w/o using larger more vertical tech (e.g. Hadoop, etc) Taken on larger data loads, it scales, can sleep at night. I can use for both data from erp & also scale needed for cloud Search, GeoSearch, HA, Different storage engines for the right behavior, Ops: Ops team is enabled for success - Right tools for the right use cases Companies/Services out there to get you up and running!