Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go
•Als POTX, PDF herunterladen•
1 gefällt mir•2,608 views
With the release of MongoDB 3.0, the tools (mongodump, mongoimport, mongotop, etc) have been completely re-designed and re-written in Go to improve maintainability and performance. In this section, we'll give an architectural overview of the tools, describe how we used Go's native capacities to improve their parallelism, and also take a deep technical dive into their internals. We'll discuss performance, usability and integration improvements and share advanced techniques for power users. With a better understanding of how the tools work, you should feel comfortable effectively using and contributing to the tools.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go
Ähnlich wie Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go (20)
Mehr von MongoDB
Mehr von MongoDB (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go
- 1. Putting the Go in MongoDB Wisdom Omuya Software Engineer wisdom@mongodb.com
- 6. Export
- 10. Mongorestore Concurrency in 2.6
- 11. Invalid Documents ● Nested CSV, TSV imports { "_id": ObjectId("5542593837096bdf8bbb6d91"), “name.first”: ”Wisdom”, ”name.last”: ”Omuya" } name.first,name.last Wisdom,Omuya
- 12. mongo> db.test.find({“name.first":”Wisdom"}) Fetched 0 record(s) in 0ms Unretrievable Data { "_id": ObjectId("5542593837096bdf8bbb6d91"), “name.first”: ”Wisdom”, ”name.last”: ”Omuya" }
- 14. Loose Coupling
- 16. Portability
- 18. General ● Storage engine support ● Wiredtiger, RocksDB, etc ● Backwards compatibility ● Dump and restore BSON ● Import and export JSON
- 19. ● Excellent support for concurrency ● Runs on all supported platforms ● Easier to write and debug ● Fun! Why Go?
- 21. mongoimport
- 23. New Mongoimport Flags --numDecodingWorkers (autodetect to max) --numInsertionWorkers (default 1) --batchSize (default 10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)
- 24. Input Validation ● No more broken imports, illegal field names ”a”, “b.”
- 25. Dot-nesting ● Nested CSV, TSV imports name.first,name.last Wisdom,Omuya { "_id": ObjectId("55425a3c37096bdf8bbb6d93"), "name": { "first": ”Wisdom", "last": ”Omuya" } }
- 27. Write Concern Specificity ● New default w=majority on import/restore/files o safer, matches what our users assume is happening ● --writeConcern flag e.g. ‘{w: 3, j: true, fsync: false, wtimeout: 400}’
- 28. Mongoimport Concurrency in 3.0
- 31. New Mongorestore Flags --numParallelCollections (aka -j default=4) --numInsertionWorkersPerCollection (default=1) --batchSize(default 10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)
- 32. Mongorestore Concurrency in 3.0 2X speedup 4X speedup
- 33. What’s next? • mongorestore and mongodump now support compression and archiving • No need for directories containing BSON files mongodump -d weather -h localhost --archive --gzip | mongorestore -h remotehost --archive --gzip
- 34. Resources • Repository: https://github.com/mongodb/mongo-tools • Usage Guide: https://docs.mongodb.org/master/reference/program/ • Issue Tracking: https://jira.mongodb.org/browse/tools/
- 35. THANKS! tools
Hinweis der Redaktion
- Don’t say ‘kind of’!
- First release for MongoDB tools in Go Propose a thesis: we rewrote it, it's much faster and you'll be a more effective user because of the concurrency and control we've introuced
- First release for MongoDB tools in Go We rewrote it, it's much faster and you'll be a more effective user because of the concurrency and control we've introuced
- Mongofiles usually files above our 16MB limit
- Stat: Collects statistics of your mongod’s; Similar to vmstat or iostat Top: Tracks the amount of time spent per operation on different namespaces
- aka, mon-goop-log :)
- why couldn't we multithread the old version of the database? 12 core Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz
- Old tool didn’t try and prevent you from shooting yourself in foot
- Unfindable imports
- 24MB -> 6MB In order to parse the query a chunk of the query engine, all of boost We can iterate faster
- easier for the community to contribute to the tools easier to iterate separately
- Emphasize concurrency and control
- quick growth of the team, onboarding
- Mongofiles usually files above our 16MB limit
- Short anecdote? Tool gains not server gains batch size is # of docs
- One design principle is we didn't want to potentially overwhelm a mongodb server when ppl are used to single threaded
- As much nesting as you want
- Full JSON roundtripping
- No concurrency controls in 2.6 23GB <-> 11 collections Emphasize concurrency and control
- cheap way of making asynchronous writes
- No concurrency controls in 2.6 23GB <-> 11 collections Emphasize concurrency and control
- Not write to disk at al -> separate CPU Compressed network archive archive support is not in 3.0, but initially available in 3.1.x