Replacing Oracle with MongoDB for a templating application at the Bavarian government

MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing

Christian BrensingBayerisches Landesamt fürStatistik und DatenverarbeitungRechenzentrum Süd

• Template-Processor (~ JSP)• ODF and RTF• PDF / PostScript postprocessing• Groovy, Python, Ruby, Tcl• In Production using RDBMS since 2008• Central HR system (SAP) uses BayText to create output

Client for developing the templates (BayText-IDE)

ODF-Template using Ruby

Generated document

2 MillionGenerated documents per year

Tiny dataset50,000 records and 10 GB BLOB

a small system after all

So why migrate?

Because we can

developers.pop() Smaller teams

Easy to learnRead one book and youre done

Simple mapping

permissions templates documentscodes groups document_usagesfolder_conﬁgurations requests foldersnodes structures template_tagsdynaﬁelds components roles_rolesroles bundles scripts

nodes rolesproperties GridFS

{! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}

public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...)); Easy mapping using Spring-ConversionService

Performance up to 10x

ORM no moreThe MongoDB driver is all you need

Obstacles?

Mentality

Referential Integrity Multi-Document-Update

{! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!} Linking documents via a path instead of ObjectId

But paths are mutable! All references must be updated

Approach• Pseudo-Transaction with an update locking mechanism in a custom oplog collection• Periodical repair jobs ﬁnishing cancelled operations• Extended interpretation of eventual consistency

Unit-TestsContinuous Integration

F.I.R.S.T.

IsolatedTests are using a unique DB per JVM

@Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Test-DB löschen! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf(-);! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!} Test-ApplicationContext

RepeatableCollections are purged before each test method

public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.remove(new BasicDBObject());! }! }! }!} Spring-TestExecutionListener

@RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}! Test base class

95% Coverage

Summary• Developing the data layer is fun again• Reduced complexity• Flat learning curve compared to SQL/ORM• use_mongo() unless transactions_required

Thank you!

https://twitter.com	51
https://web.tweetdeck.com	1
http://gazeta.yandex.ru	1
http://tweetedtimes.com	1

Replacing Oracle with MongoDB for a templating application at the Bavarian government

von comsysto am 19. September 2012

Zugänglichkeit

Kategorien

Details hochladen

Benutzerrechte

4 Einbettungen 54

Statistiken

Replacing Oracle with MongoDB for a templating application at the Bavarian government Presentation Transcript