1. 2013-07-05 Europython Florence mongopersist 1/37
mongopersist
Adam GROSZER
https://github.com/zopefoundation/mongopersist
https://pypi.python.org/pypi/mongopersist
slideshare URL
https://github.com/agroszer
http://hu.linkedin.com/in/agroszer/

2. 2013-07-05 Europython Florence mongopersist 2/37
contents
● mongoDB
● databases
● persistence
● mongopersist features

3. 2013-07-05 Europython Florence mongopersist 3/37
mongopersist

4. 2013-07-05 Europython Florence mongopersist 4/37

5. 2013-07-05 Europython Florence mongopersist 5/37
choose your database, choose your future
pro con
ZODB
- very transparent
- object store
- only python/native
- no query lang
- no 3rd
party tools
- no default indexing
RDBMS
ORM
- ad-hoc SQL queries
- indexes, tools, etc.
- BIG impedance
mismatch
- limited transparency
- strict schema
mongoDB
- document store
- ad-hoc queries
- indexes, tools, etc.
- small impedance
mismatch
- limited/no
transparency

6. 2013-07-05 Europython Florence mongopersist 6/37
persistence
state that outlives the process
- get the object
- modify
- (finish the transaction)

7. 2013-07-05 Europython Florence mongopersist 7/37
persistence
hand-crafted example, using pymongo:
>>> stephan = db.coll.find_one(
... {'name': 'Stephan'})
>>> stephan['phone'] = {
... 'country': '+1',
... 'area': '555',
... 'number': '3945456'}
>>> db.coll.save(stephan)

8. 2013-07-05 Europython Florence mongopersist 8/37
persistence
ideal case with mongopersist:
>>> persons = dm.root
>>> stephan = persons['stephan']
>>> stephan.phone = Phone(
... ‘+1’, ‘555’, ‘3945456’)
>>> transaction.commit()

9. 2013-07-05 Europython Florence mongopersist 9/37
mongopersist features
● transparency
● transactions Optimistic Data Dumping
● write conflict detection
● custom de/serialization
● object caching
● query logging, incl. traceback
● (not just) ZTK corner

10. 2013-07-05 Europython Florence mongopersist 10/37
sample class
class Person(persistent.Persistent):
def __init__(self, name, phone=None,
address=None, friends=None,
visited=(), birthday=None):
self.name = name
self.address = address
self.friends = friends or {}
self.visited = visited
self.phone = phone
self.birthday = birthday
self.today = datetime.datetime.now()
...

11. 2013-07-05 Europython Florence mongopersist 11/37
transparency
setup code:
>>> conn = pymongo.Connection(
... 'localhost', 27017, tz_aware=False)
>>> from mongopersist import datamanager
>>> dm =
datamanager.MongoDataManager(conn)

12. 2013-07-05 Europython Florence mongopersist 12/37
transparency
>>> dm.root
>>> stephan = Person(u'Stephan')
>>> dm.root['stephan'] = stephan
>>> stephan = dm.root['stephan']
>>> stephan.name = u'Stephan Richter'
>>> transaction.commit()

13. 2013-07-05 Europython Florence mongopersist 13/37
defaults
>>> dumpCollection('__main__.Person')
[{u'_id': ObjectId('...'),
u'address': None,
u'birthday': None,
u'friends': {},
u'name': u'Stephan Richter',
u'phone': None,
u'today': datetime.datetime(2013, 6, 18, 14,
48, 30, 970000),
u'visited': []}]

14. 2013-07-05 Europython Florence mongopersist 14/37
customizing
class Address(persistent.Persistent):
_p_mongo_collection = 'address'
def __init__(self, city, zip):
self.city = city
self.zip = zip

15. 2013-07-05 Europython Florence mongopersist 15/37
sub-objects via DBRef
>>> stephan.address = Address('Maynard', '01754')
>>> transaction.commit()
>>> dumpCollection('address')
[{u'_id': ObjectId('...'),
u'city': u'Maynard',
u'zip': u'01754'}]
>>> dumpCollection('__main__.Person')
[{u'_id': ObjectId('...'),
u'address': DBRef(u'address', ObjectId('...'),
u'mongopersist_test'), ...}]

16. 2013-07-05 Europython Florence mongopersist 16/37
collection sharing
class Person(persistent.Persistent):
_p_mongo_collection = 'person'
name = u''
...
class Employee(Person):
_p_mongo_collection = 'person'
salary = 0
...
mongopersist will automatically notice these cases and
stores the Python type as part of the document

17. 2013-07-05 Europython Florence mongopersist 17/37
sub object/document
class Car(persistent.Persistent):
_p_mongo_sub_object = True
def __init__(self, year, make, model):
self.year = year
self.make = make
self.model = model
>>> dm.root['stephan'].car = Car('2005', 'Ford', 'Explorer')
>>> dumpCollection('__main__.Person')
[{...
u'car': {u'_py_persistent_type': u'__main__.Car',
u'make': u'Ford',
u'model': u'Explorer',
u'year': u'2005'}, ...}]

18. 2013-07-05 Europython Florence mongopersist 18/37
beware of non Persistent objects
class Phone(object):
def __init__(self, country, area, number):
...
>>> stephan.phone = Phone('+1', '978', '394-5124')
>>> dumpCollection('__main__.Person')
[{...
u'phone': {u'_py_type': u'__main__.Phone',
u'area': u'978',
u'country': u'+1',
u'number': u'394-5124'}, ...]
>>> stephan.phone.number = '555-1234'
>>> transaction.commit()
Changes not saved, because not subclassing Persistent

19. 2013-07-05 Europython Florence mongopersist 19/37
add/delete property
>>> stephan.foobar = 42
>>> transaction.commit()
No declaration needed!
>>> dumpCollection('__main__.Person')
[{...
u'foobar': 42,
u'name': u'Stephan',
...}]
>>> del stephan.foobar
>>> transaction.commit()

20. 2013-07-05 Europython Florence mongopersist 20/37
custom property
>>> stephan.friends[u'roger'] = Person(u'Roger')
>>> stephan.visited.append('Italy')
>>> transaction.commit()
>>> dumpCollection('__main__.Person')
[{...
u'friends': {u'roger': DBRef(u'__main__.Person',
ObjectId('...'), u'mongopersist_test')},
u'visited': [u'Italy']},
{... u'name': u'Roger', ...]
>>> stephan.friends[u'roger'].name
u'Roger'
Circular references:
- Persistent: OK
- non-Persistent: no-go!

21. 2013-07-05 Europython Florence mongopersist 21/37
Optimistic Data Dumping
The process of dumping data during a transaction
under the assumption the transaction will
succeed.
object modifications
...
object modifications
...
automatic/implicit flush
query

22. 2013-07-05 Europython Florence mongopersist 22/37
Optimistic Data Dumping
>>> stephan.foobar = 42
...code...
>>> roy.foobar = 88
...code...
>>> dm.get_collection_from_object(
... roy).count({'foobar': 88})
1
ALL query methods are wrapped to call flush
first

23. 2013-07-05 Europython Florence mongopersist 23/37
write conflict detection
_py_serial
● Conflict detection
● Conflict resolution
Handlers:
● NoCheckConflictHandler
● Ignore conflicts, last flush wins, default
● SimpleSerialConflictHandler
● Detects conflicts, always raise ConflictError
● ResolvingSerialConflictHandler
● Detects conflicts, calls _p_resolveConflict()

24. 2013-07-05 Europython Florence mongopersist 24/37
custom serializers
stephan.birthday = datetime.date(1980, 1, 25)
u'birthday': {u'_py_factory': u'datetime.date',
u'_py_factory_args': [Binary('x07xbcx01x19', 0)]},
class DateSerializer(serialize.ObjectSerializer):
def can_read(self, state):
return isinstance(state, dict) and
state.get('_py_type') == 'datetime.date'
def read(self, state):
return datetime.date.fromordinal(state['ordinal'])
def can_write(self, obj):
return isinstance(obj, datetime.date)
def write(self, obj):
return {'_py_type': 'datetime.date', 'ordinal': obj.toordinal()}
>>> serialize.SERIALIZERS.append(DateSerializer())
u'birthday': {u'_py_type': u'datetime.date', u'ordinal': 722839},

25. 2013-07-05 Europython Florence mongopersist 25/37
querying mongoDB
datamanager.get_collection(dbname, collname)
datamanager.get_collection_from_object(obj)
- find, find_one, count, etc
extra methods, which return objects:
- find_objects()
- find_one_object()
ALL query methods are wrapped to call flush first
datamanager.load(dbref)

26. 2013-07-05 Europython Florence mongopersist 26/37
object caching
DB access and object instantiation is quite slow
● class Lookup Cache: dbref → python class
lookup
● object Cache: dbref → object (within
transaction)
● document Cache: dbref → document (avoid DB
trip)

27. 2013-07-05 Europython Florence mongopersist 27/37
query logging, incl. traceback
LoggingDecorator: Logs the calls to
insert, update, remove, save, find, find_one,
find_and_modify, count
including args and kwargs.
With optional traceback.

28. 2013-07-05 Europython Florence mongopersist 28/37
containers and collections
class People(MongoCollectionMapping):
__mongo_collection__ = '__main__.Person'
__mongo_mapping_key__ = 'name'
● Mapping/dict API for a Mongo collection.
● Specify the collection to use for the mapping
● Specify the attribute that represents the
dictionary key.

29. 2013-07-05 Europython Florence mongopersist 29/37
Contained, Container, __name__, __parent__
zope.container.MongoContained
zope.container.MongoContainer
- mapping/dict interface
- and more: add, (raw_)find, (raw_)find_one
zope.container.IdNamesMongoContainer
- uses the item's ObjectID as key
(not just) ZTK corner

30. 2013-07-05 Europython Florence mongopersist 30/37
(not just) ZTK corner
connection pooling
app setup:
mdmp = pool.MongoDataManagerProvider(
host='localhost', port=27017,
logLevel=20, tz_aware=True, w=1, j=True)
zope.component.provideUtility(mdmp)
request setup:
mdmp = getUtility(interfaces.IMongoDataManagerProvider)
dm = mdmp.get()

31. 2013-07-05 Europython Florence mongopersist 31/37
(not just) ZTK corner
IMongoAttributeAnnotatable
- storing metadata
DublinCore
- standard for metadata

32. 2013-07-05 Europython Florence mongopersist 32/37
Q&A

33. 2013-07-05 Europython Florence mongopersist 33/37
timesink

34. 2013-07-05 Europython Florence mongopersist 34/37
pickle
persistence in python=pickle?

35. 2013-07-05 Europython Florence mongopersist 35/37
databases
pro con
ZODB
- very transparent
- object store
- only python/native
- no query lang
- no 3rd
party tools
- no default indexing
RDBMS
ORM
- ad-hoc SQL queries
- indexes, tools, etc.
- BIG impedance
mismatch
- limited transparency
- rigid schema
mongoDB
- document store
- ad-hoc queries
- indexes, tools, etc.
- small impedance
mismatch
- limited/no
transparency

36. 2013-07-05 Europython Florence mongopersist 36/37
databases
pro con
ZODB
- very transparent
- object store
- only python/native
- no query lang
- no 3rd
party tools
- no default indexing
RDBMS
ORM
- ad-hoc SQL queries
- indexes, tools, etc.
- BIG impedance
mismatch
- limited transparency
- strict schema
mongoDB
- document store
- ad-hoc queries
- indexes, tools, etc.
- small impedance
mismatch
- limited/no
transparency

37. 2013-07-05 Europython Florence mongopersist 37/37
databases
pro con
ZODB
- very transparent
- object store
- only python/native
- no query lang
- no 3rd
party tools
- no default indexing
RDBMS
ORM
- ad-hoc SQL queries
- indexes, tools, etc.
- BIG impedance
mismatch
- limited transparency
- rigid schema
mongoDB
- document store
- ad-hoc queries
- indexes, tools, etc.
- small impedance
mismatch
- limited/no
transparency