SlideShare wird heruntergeladen. ×
0
MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing
Christian BrensingBayerisches Landesamt fürStatistik und DatenverarbeitungRechenzentrum Süd
• Template-Processor (~ JSP)• ODF and RTF• PDF / PostScript postprocessing• Groovy, Python, Ruby, Tcl• In Production using...
Client for developing the templates (BayText-IDE)
ODF-Template using Ruby
Generated document
2 MillionGenerated documents per year
Tiny dataset50,000 records and 10 GB BLOB
a small system after all
So why migrate?
Because we can
developers.pop()     Smaller teams
Easy to learnRead one book and youre done
Simple mapping
permissions       templates           documentscodes       groups           document_usagesfolder_configurations       requ...
nodes  rolesproperties GridFS
{!     "_id" : ObjectId(),!     "_type" : "Document",!     "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),!     "updat...
public class RoleReadConverter implements Converter<DBObject, Role> {!    @Override!    public Role convert(DBObject sourc...
Performance   up to 10x
ORM no moreThe MongoDB driver is all you need
Obstacles?
Mentality
Referential Integrity     Multi-Document-Update
{!     "_id" : ObjectId(),!     "_type" : "Folder", !     "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),!     "update...
But paths are mutable!    All references must be updated
Approach• Pseudo-Transaction with an update locking  mechanism in a custom oplog collection• Periodical repair jobs finishi...
Unit-TestsContinuous Integration
F.I.R.S.T.
IsolatedTests are using a unique DB per JVM
@Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {!    @Override!    publi...
RepeatableCollections are purged before each test method
public class MongoTestExcecutionListener extends AbstractTestExecutionListener {!    @Override!    public void beforeTestM...
@RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration....
95% Coverage
Summary• Developing the data layer is fun again• Reduced complexity• Flat learning curve compared to SQL/ORM• use_mongo() ...
Thank you!
Nächste SlideShare
Wird geladen in ...5
×

Replacing Oracle with MongoDB for a templating application at the Bavarian government

1,308

Published on

Bavarian government runs a document template application (RTF or ODF with Groovy, Python, Ruby or Tcl as scripting language) serving different government offices. Having complex and hierarchical data structures to organize the templates, MongoDB was selected to replace the Oracle-based persistence layer. This presentation is about the improvements they have achieved with the migration to MongoDB, problems they had to solve underway and unit testing of the persistence layer in order to keep their quality level. Presentation slides by Christian Brensing, Senior Developer at Rechenzentrum Süd, shown at Munich MongoDB User Group Meetup on 18th September 2012

Published in: Technologie, Business
0 Kommentare
2 Gefällt mir
Statistiken
Notizen
  • Hinterlassen Sie den ersten Kommentar

Keine Downloads
Views
Gesamtviews
1,308
Bei Slideshare
0
Aus Einbettungen
0
Anzahl an Einbettungen
3
Aktionen
Geteilt
0
Downloads
9
Kommentare
0
Gefällt mir
2
Einbettungen 0
No embeds

No notes for slide

Transcript of "Replacing Oracle with MongoDB for a templating application at the Bavarian government"

  1. 1. MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing
  2. 2. Christian BrensingBayerisches Landesamt fürStatistik und DatenverarbeitungRechenzentrum Süd
  3. 3. • Template-Processor (~ JSP)• ODF and RTF• PDF / PostScript postprocessing• Groovy, Python, Ruby, Tcl• In Production using RDBMS since 2008• Central HR system (SAP) uses BayText to create output
  4. 4. Client for developing the templates (BayText-IDE)
  5. 5. ODF-Template using Ruby
  6. 6. Generated document
  7. 7. 2 MillionGenerated documents per year
  8. 8. Tiny dataset50,000 records and 10 GB BLOB
  9. 9. a small system after all
  10. 10. So why migrate?
  11. 11. Because we can
  12. 12. developers.pop() Smaller teams
  13. 13. Easy to learnRead one book and youre done
  14. 14. Simple mapping
  15. 15. permissions templates documentscodes groups document_usagesfolder_configurations requests foldersnodes structures template_tagsdynafields components roles_rolesroles bundles scripts
  16. 16. nodes rolesproperties GridFS
  17. 17. {! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}
  18. 18. public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...)); Easy mapping using Spring-ConversionService
  19. 19. Performance up to 10x
  20. 20. ORM no moreThe MongoDB driver is all you need
  21. 21. Obstacles?
  22. 22. Mentality
  23. 23. Referential Integrity Multi-Document-Update
  24. 24. {! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!} Linking documents via a path instead of ObjectId
  25. 25. But paths are mutable! All references must be updated
  26. 26. Approach• Pseudo-Transaction with an update locking mechanism in a custom oplog collection• Periodical repair jobs finishing cancelled operations• Extended interpretation of eventual consistency
  27. 27. Unit-TestsContinuous Integration
  28. 28. F.I.R.S.T.
  29. 29. IsolatedTests are using a unique DB per JVM
  30. 30. @Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Test-DB löschen! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf(-);! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!} Test-ApplicationContext
  31. 31. RepeatableCollections are purged before each test method
  32. 32. public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.remove(new BasicDBObject());! }! }! }!} Spring-TestExecutionListener
  33. 33. @RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}! Test base class
  34. 34. 95% Coverage
  35. 35. Summary• Developing the data layer is fun again• Reduced complexity• Flat learning curve compared to SQL/ORM• use_mongo() unless transactions_required
  36. 36. Thank you!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×