Replacing Oracle with MongoDB for a templating application at the Bavarian government
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Replacing Oracle with MongoDB for a templating application at the Bavarian government

am

  • 1,718 Views

Bavarian government runs a document template application (RTF or ODF with Groovy, Python, Ruby or Tcl as scripting language) serving different government offices. Having complex and hierarchical data ...

Bavarian government runs a document template application (RTF or ODF with Groovy, Python, Ruby or Tcl as scripting language) serving different government offices. Having complex and hierarchical data structures to organize the templates, MongoDB was selected to replace the Oracle-based persistence layer. This presentation is about the improvements they have achieved with the migration to MongoDB, problems they had to solve underway and unit testing of the persistence layer in order to keep their quality level. Presentation slides by Christian Brensing, Senior Developer at Rechenzentrum Süd, shown at Munich MongoDB User Group Meetup on 18th September 2012

Statistiken

Views

Gesamtviews
1,718
Views auf SlideShare
1,644
Views einbetten
74

Actions

Gefällt mir
2
Downloads
5
Kommentare
0

4 Einbettungen 74

https://twitter.com 71
https://web.tweetdeck.com 1
http://gazeta.yandex.ru 1
http://tweetedtimes.com 1

Zugänglichkeit

Kategorien

Details hochladen

Uploaded via as Adobe PDF

Benutzerrechte

© Alle Rechte vorbehalten

Report content

Als unangemessen gemeldet Als unangemessen melden
Als unangemessen melden

Wählen Sie Ihren Grund, warum Sie diese Präsentation als unangemessen melden.

Löschen
  • Full Name Full Name Comment goes here.
    Sind Sie sicher, dass Sie...
    Ihre Nachricht erscheint hier
    Processing...
Kommentar posten
Kommentar bearbeiten

Replacing Oracle with MongoDB for a templating application at the Bavarian government Presentation Transcript

  • 1. MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing
  • 2. Christian BrensingBayerisches Landesamt fürStatistik und DatenverarbeitungRechenzentrum Süd
  • 3. • Template-Processor (~ JSP)• ODF and RTF• PDF / PostScript postprocessing• Groovy, Python, Ruby, Tcl• In Production using RDBMS since 2008• Central HR system (SAP) uses BayText to create output
  • 4. Client for developing the templates (BayText-IDE)
  • 5. ODF-Template using Ruby
  • 6. Generated document
  • 7. 2 MillionGenerated documents per year
  • 8. Tiny dataset50,000 records and 10 GB BLOB
  • 9. a small system after all
  • 10. So why migrate?
  • 11. Because we can
  • 12. developers.pop() Smaller teams
  • 13. Easy to learnRead one book and youre done
  • 14. Simple mapping
  • 15. permissions templates documentscodes groups document_usagesfolder_configurations requests foldersnodes structures template_tagsdynafields components roles_rolesroles bundles scripts
  • 16. nodes rolesproperties GridFS
  • 17. {! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}
  • 18. public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...)); Easy mapping using Spring-ConversionService
  • 19. Performance up to 10x
  • 20. ORM no moreThe MongoDB driver is all you need
  • 21. Obstacles?
  • 22. Mentality
  • 23. Referential Integrity Multi-Document-Update
  • 24. {! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!} Linking documents via a path instead of ObjectId
  • 25. But paths are mutable! All references must be updated
  • 26. Approach• Pseudo-Transaction with an update locking mechanism in a custom oplog collection• Periodical repair jobs finishing cancelled operations• Extended interpretation of eventual consistency
  • 27. Unit-TestsContinuous Integration
  • 28. F.I.R.S.T.
  • 29. IsolatedTests are using a unique DB per JVM
  • 30. @Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Test-DB löschen! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf(-);! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!} Test-ApplicationContext
  • 31. RepeatableCollections are purged before each test method
  • 32. public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.remove(new BasicDBObject());! }! }! }!} Spring-TestExecutionListener
  • 33. @RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}! Test base class
  • 34. 95% Coverage
  • 35. Summary• Developing the data layer is fun again• Reduced complexity• Flat learning curve compared to SQL/ORM• use_mongo() unless transactions_required
  • 36. Thank you!