Presentation on Schema Service for harnessing MongoDB schemaless feature. Implemented using Navajo Service Oriented Application Framework. Digital Match Report of the Sportlink ERP system as use case.
3. Arjen Schoneveld
⢠co-Founder of Dexels,Amsterdam based
software company
⢠Architect of Sportlink Sports ERP system
⢠PhD in Computer Science
⢠arjen@dexels.com, http://nl.linkedin.com/in/
aschoneveld/
Friday, August 23, 13
4. Our Story
⢠How it started: Sportlink Sport Management System
⢠Navajo: a Service Oriented Application Framework
⢠Limitations of monolithic persistence
⢠MongoDB use case: Sportlink Digital Match Report
⢠Using Navajo Entity Services as a Schema
⢠RESTfull Entities
Friday, August 23, 13
5. Sportlink Architecture (2003)
Sport Club
Application
Federation
Application
Federation
Website
Database (Oracle) Crystal
Reports
Comp.
Planning
Friday, August 23, 13
6. Data Model
Object Model
Service Facade
Web Services
JSP/HTML/
Javascript/CSS
ORM
JAX-WS
SOAP/WSDL
Service Proxy
A Service Oriented Application Stack
Object Model
Friday, August 23, 13
8. Services First
⢠Changes are âthe ruleâ in many environments
⢠Most Services are CRUD and instances have
very short live-span
⢠Many services do not need âOOP modelsâ
⢠Use OOP for complex algorithms (e.g. match
scheduling) only
⢠Build, not generate, Services
Friday, August 23, 13
9. Usable Services
⢠Loose Coupling: no client stub generation!
⢠Metadata is key ingredient: self describing
⢠Type, description, input validation, subsequent
services (protocol)
⢠Self describing Abstract User Interface â enables
automatic creation of usable ďŹrst order UIs
⢠Use a Canonical Service Language as the âlingua
francaâ between the services and the client
Friday, August 23, 13
11. Data Model
The Navajo Application Framework
UIs
[Platform Agnostic
Declarative UI Language]
Canonical
Service
Language
Services
[Service Script Language]
Friday, August 23, 13
15. Sportlink Ecosystem (2013)
Club
(Tipi/Swing)
Association
(Java/Swing)
Website
(Third Party)
Oracle
Crystal
FTP
~4000 services
Comp.
Planning
E-Mail
Digital
Match Report
(iOS/Android/Tipi)
Internet
Payment
(PHP)
Facility Mngtmt
(GWT+G. Earth)
Course
Registration
(GWT+G. Maps)
OfďŹcial Portal
(Tipi/VaadIn)
MongoDB
MySQL
Website
Plugins
(PHP)
Financial
Admin
ServiceWorkďŹow
Enterprise
DB
Friday, August 23, 13
16. Growing Pains
⢠More end user applications
⢠More functionality
⢠More sport associations with different
requirements
⢠More end users
⢠Single database has become the bottleneck
Friday, August 23, 13
17. Oracle Limitations
⢠License Costs!
⢠Unpredictable query behavior
⢠Only partly/expensive High Availability
⢠Poor affordable(!) horizontal scalability in
combination with good single node
performance
⢠Service/Persistence impedance mismatch!
Friday, August 23, 13
19. Our MongoDB Use Case
⢠Digital Match Report (Mobile,Web and
Desktop)
⢠Player Roster for home- and away teams
(role, captain, base player)
⢠Match Result, Match Events, Player
Validation, Digital Players Pass
⢠40.000 matches per weekend (ďŹxed times:
11.00, 12:30, 14:00, 15:30, ...)
⢠typical: 3 users per match (ofďŹcial, two
captains)
Friday, August 23, 13
20. Why MongoDB
⢠MatchReport application is introduced
gradually â Scalability â NoSQL
⢠MatchReport can be fetched and updated
like a Data Aggregate â Document Based
NoSQL
Friday, August 23, 13
21. Home Away OfďŹcial Mongo Oracle
Fetch Match Report
MatchReportReport Fetch Match Report
Report
Populate Home Players Report
Populate Away Players Report
Fetch Match Report
Report
Report Report
Fill In Report Report
Finalize Report Report Match
Players
Events
Players
Events
Friday, August 23, 13
22. How do we integrate Navajo and
MongoDB?
Friday, August 23, 13
23. Navajo Services
⢠A simple XML/HTTP/POST-based Web Service
⢠Uses a Canonical Service Language: Navajo Document
⢠Consumes and produces a Navajo Document (Everything is a Document)
⢠May contain additional (Service) links to next-step services (implementing an
application protocol, like hyperlinks)
⢠Navascript consumes and produces a Navajo Document, using expressions
and access (Java) adapters for: database access, calling other Navajo
Services, calling SOAP, manipulating messages (e.g. joins), etc.
⢠JavaScript and JRuby are also supported Navajo Service implementation
languages
⢠Navajo Services are (hot) deployed in a Navajo Server/OSGi runtime using
realtime generation of OSGi bundles
Friday, August 23, 13
24. Navajo Document
⢠Navajo Services use Navajo Documents for all
message processing (Canonical Service Language)
⢠All data required by and delivered from a Navajo
Service is contained in a Navajo Document
⢠A Navajo document consists of the following
elements:
⢠messages, for grouping:
⢠properties and other messages
⢠methods (links to relevant subsequent services)
Friday, August 23, 13
25. Properties
⢠A property has metadata:
⢠name. Name must be unique within a message
⢠type. Natively supported types are: string, integer, ďŹoat, boolean, date,
binary, selections(!).
⢠value.The value of the property.
⢠description. Used to indicate the semantical meaning of the property
⢠subtype. May contain additional formatting hints for clients, like matching
regular expressions for strings or the mime-type of a binary property
⢠direction. Indication for a client to render a property read-only or not.
Friday, August 23, 13
26. Serializations
⢠The default serialization of Navajo Document is XML
⢠Different other serialization are supported, e.g. JSON
⢠Navajo Document contains a lot of metadata
automatic User Interface Generation is supported (an
Eclipse Plugin is provided)
⢠Serialization of binary type properties is Base64.
Navajo XML parser supports transparent streaming of
binary properties
Friday, August 23, 13
27. A Navajo Document<tml>
<message name=âPersonâ>
<property name=âLastnameâ length=â100â description=âLastname of the personâ type=âstringâ
subtype=âregexp=[A-z]*â direction=âoutâ value=âKantâ/>
<property name=âFirstNameâ type=âstringâ subtype=âregexp=[A-z]*â length=â50â direction=âoutâ
value=âImmanuelâ/>
<property name=âDateOfBirthâ type=âdateâ direction=âoutâ value=â1724-04-22â/>
<property name=âPhotoâ type=âbinaryâ direction=âinâ>
</property>
</message>
</tml>
Friday, August 23, 13
32. MongoDB
⢠MongoDB is a document-based NoSQL database
⢠MongoDB is schema-less, unlike SQL databases that
rely on a predeďŹned schema (or metadata) to deďŹne
e.g. tables and table structures
⢠MongoDB serializes data as JSON objects
⢠MongoDB supports a sharded setup â scalability
⢠Advantage: ease of use due to lack of structure
⢠Disadvantage: unmaintainable due to lack of structure
Friday, August 23, 13
33. Navajo and MongoDB
⢠Store Navajo Document directly in
MongoDB
⢠Navajo Document can be easily
transformed to JSON
⢠No need for writing âmapping codeâ (no
impedance mismatch)
Friday, August 23, 13
34. JSONify Navajo Document
<message name=âMatchReportâ>
<property name=âMatchIdâ value=â1â type=âintegerâ/>
<property name=âHomeTeamIdâ value=â1234â type=âintegerâ description=âUnique id of the home teamâ/>
<property name=âAwayTeamIdâ value=â4321â type=â integerâ description=âUnique id of the away teamâ/>
<property name=âDateâ value=â2013-08-24â type=âdateâ description=âMatch dateâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=âHomePlayersâ type=âarrayelementâ>
<property name=âPersonIdâ type=âstringâ value=âCHGP12Yâ/>
<property name=âRoleâ type=âselectionâ cardinality=â1â type=âselectionâ>
<option name=âDefenderâ value=âDEFâ selected=â1â/>
<option name=âKeeperâ value=âKPRâ selected=â0â/>
</property>
</message>
</message>
[... same for AwayPlayers ... left for brevity ]
</message>
The Navajo Document XML above can be converted to JSON as follows:
{ âMatchIdâ : 1,âHomeTeamIdâ :â1234â,âAwayTeamIdâ :â4321â, âDateâ : Date ( 23242873 ), âHomePlayersâ :
[ {âPersonIdâ :âCHGP12Yâ,âRoleâ :âDEFâ}] }
Friday, August 23, 13
35. Query MongoDB
⢠Entity in MongoDB can be queried using MongoDB query syntax
⢠Entity in MongoDB can be queried using a Navajo Document -> will be converted
to a JSON query:
<message name=âMatchReportâ>
<property name=âMatchIdâ value=â1â/>
</message>
can be converted to Mongo query (message name maps to Collection name):
MatchReport.ďŹnd({âMatchIdâ : 1})
Friday, August 23, 13
36. <navascript>
<message name=âMatchReportâ>
<property name=âMatchIdâ value=â1â type=âintegerâ/>
</message>
<map.navajomap useCurrentOutDoc=âtrueâ>
<navajomap.callwebservice name=âmongo/ProcessQueryMongoâ append=ââ/ââ/>
</map.navajomap>
</navascript>
Mongo Query Service
Construct Query
Call Query Service and append result
Friday, August 23, 13
37. Result of Query
<message name=âMatchReportâ>Unique Id
<property name=â_idâ value=âaabf3fdde323gâ/>
<property name=âMatchIdâ value=â1â type=âintegerâ/>
<property name=âHomeTeamIdâ value=â1234â type=âintegerâ/>
<property name=âAwayTeamIdâ value=â4321â type=âintegerâ/>
<property name=âDateâ value=â2013-08-24â type=âdateâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=âHomePlayersâ>
<property name=â__idâ value=âabb344gf67â/>
<property name=âPersonIdâ type=âstringâ value=âCHGP12Yâ/>
<property name=âRoleâ type=âstringâ value=âDEFâ/>
</message>
<message name=âHomePlayersâ>
<property name=â__idâ value=âffhggab3232bâ/>
<property name=âPersonIdâ type=âstringâ value=âMJGP845â/>
<property name=âRoleâ type=âstringâ value=âKEPâ/>
</message>
</message>
</message>
What is Lost?
What is Added?
Unique Id
Unique Id
Friday, August 23, 13
38. How can we resolve this?
Friday, August 23, 13
39. Query using Entity
Service
⢠DeďŹne a Navajo Service for producing the Entity
Navajo Document: Entity Service as a Schema
⢠Expects no input
⢠Simply returns Navajo Document describing the
Entity
⢠Pass Entity Service as a âparameterâ to the Navajo
Service that queries or mutates Mongo
⢠Entity service is used to decorate the ârawâ Mongo
document
Friday, August 23, 13
40. Example Entity Message
<message name=âMatchReportâ>
<property name=âMatchIdâ type=âintegerâ/>
<property name=âHomeTeamIdâ type=âintegerâ description=âUnique id of the
hometeamâ/>
<property name=âAwayTeamIdâ type=âintegerâ description=âUnique id of the
personâ/>
<property name=âDateâ type=âdateâ description=âMatch dateâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=â HomePlayersâ type=âdeďŹnitionâ>
<property name=âPersonIdâ type=âstringâ value=âUnique person idâ/>
<property name=âRoleâ type=âselectionâ cardinality=â1â type=âselectionâ>
<option name=âDefenderâ value=âDEFâ/>
<option name=âKeeperâ value=âKPRâ/>
</property>
</message>
</message>
</message>
Navajo Document above describes the Entity type âMatchReportâ
Friday, August 23, 13
42. Result of Query (revisited)
<message name=âMatchReportâ>
<property name=â_idâ value=âaabf3fdde323gâ/>
<property name=âMatchIdâ value=â1â type=âintegerâ/>
<property name=âHomeTeamIdâ value=â1234â type=âstringâ description=âUnique id of the home teamâ/>
<property name=âAwayTeamIdâ value=â4321â type=âstringâ description=â Unique id of the away teamâ/>
<property name=âDateâ value=â2013-08-24â type=âdateâ description=âMatch dateâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=â HomePlayersâ type=âdeďŹnitionâ>
<property name=âPersonIdâ type=âstringâ value=âUnique person idâ/>
<property name=âRoleâ type=âstringâ type=âselectionâ>
<option name=âDefenderâ value=âDEFâ/>
<option name=âKeeperâ value=âKPRâ/>
</property>
</message>
<message name=âHomePlayersâ>
<property name=â__idâ value=âabb344gf67â/>
<property name=âPersonIdâ type=âstringâ value=âCHGP12Yâ/>
<property name=âRoleâ type=âstringâ value=âDEFâ/>
</message>
<message name=âHomePlayersâ>
<property name=â__idâ value=âffhggab3232bâ/>
<property name=âPersonIdâ type=âstringâ value=âMJGP845â/>
<property name=âRoleâ type=âstringâ value=âKEPâ/>
</message>
</message>
</message>
Friday, August 23, 13
43. Flexibility Harnessed
⢠Entity adds missing metadata
⢠Entity can be used for type validation
⢠Adding properties is easy, simply add it to the
Entity, next time it will appear in the query
⢠Deleting properties is easy, simply delete it from
the Entity, next time it wonât appear in the query.
Friday, August 23, 13
44. Insert Example
<navascript>
<message name=â__Mongo__â>
<property name=âEntityServiceâ value=âentity/MatchReportâ/>
</message>
<message name=âMatchReportâ>
<property name=âMatchIdâ value=â2â type=âintegerâ/>
<property name=âHomeTeamIdâ value=â9787â type=âstringâ description=âUnique id of the home teamâ/>
<property name=âAwayTeamIdâ value=â6543â type=âstringâ description=â Unique id of the away teamâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=âHomePlayersâ>
<property name=âPersonIdâ type=âstringâ value=âBBFW63Xâ/>
<property name=âRoleâ type=âstringâ value=âDEFâ/>
</message>
</message>
</message>
<map.navajomap useCurrentOutDoc=âtrueâ>
<navajomap.callwebservice name=âmongo/ProcessInsertMongoâ/>
</map.navajomap>
</navascript>
This will fail!
Friday, August 23, 13
46. Delete Example
Assume that we accidentally inserted the following document:
<message name=âMatchReportâ>
<property name=â_idâ value=âaaa666â/>
<property name=âMatchIdâ value=â3â type=âintegerâ/>
<property name=âHomeTeamIdâ value=â809832â type=âintegerâ />
<property name=âAwayTeamIdâ value=â321311â type=âintegerâ />
<message name=âHomePlayersâ type=âarrayâ/>
</message>
This can be deleted using following script:
<navascript>
<message name=âMatchReportâ>
<property name=â_idâ value=âaaa666â/>
</message>
<map.navajomap useCurrentOutDoc=âtrueâ>
<navajomap.callwebservice name=âmongo/ProcessDeleteMongoâ/>
</map.navajomap>
</navascript>
Friday, August 23, 13
47. What about more ďŹne grained mutations?
Friday, August 23, 13
48. Simple Modify Example
Letâs just change the date
<navascript>
<message name=â__Mongo__â>
<property name=âEntityServiceâ value=âentity/MatchReportâ/>
</message>
<message name=âMatchReportâ>
<property name=â_idâ value=âaabf3fdde323gâ/>
<property name=âDateâ value=â2013-09-22â type=âdateâ/>
</message>
<map.navajomap useCurrentOutDoc=âtrueâ>
<navajomap.callwebservice name=âmongo/ProcessModifyMongoâ/>
</map.navajomap>
</navascript>
Friday, August 23, 13
49. Inserting new Array
Element
Insert a new player for the home team:
<message name=âMatchReportâ>
<property name=â_idâ value=âaabf3fdde323gâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=âHomePlayersâ>
<property name=âInsertâ type=âbooleanâ value=âtrueâ/>
<property name=âPersonIdâ type=âstringâ value=âBBFW06Eâ/>
<property name=âRoleâ type=âstringâ value=âDEFâ />
</message>
</message>
</message>
Note that the properties _id and Insert (set to true) are required
Id is not required,
it will be assigned
Friday, August 23, 13
50. <message name=âMatchReportâ>
<property name=â_idâ value=âaabf3fdde323gâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=âHomePlayersâ>
<property name=âModifyâ type=âbooleanâ value=âtrueâ/>
<property name=â__idâ value=âffhggab3232bâ/>
<property name=âRoleâ type=âstringâ value=âKEPâ/>
</message>
</message>
</message>
Id is required
Set Modify ďŹag
Modifying an Array
Element
Change the role of a player:
Friday, August 23, 13
51. Deleting an Array
Element
<message name=âMatchReportâ>
<property name=â_idâ value=âaabf3fdde323gâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=âHomePlayersâ>
<property name=âDeleteâ type=âbooleanâ value=âtrueâ/>
<property name=â__idâ value=âagb3f4g4468â/>
</message>
</message>
</message>
Id is required
Set Delete ďŹag
Friday, August 23, 13
52. What have we reached?
⢠A Flexible Schema to manage MongoDB
document data
⢠Schema is managed outside Mongo
⢠Basically a metadata template to ďŹlter/
validate both incoming and outgoing data
⢠No need to write any âMapping Codeâ
Friday, August 23, 13
54. What is missing?
⢠Notion of entity instances (to enable client
push)
⢠Relations between entities
⢠DeďŹnition of key/required properties
⢠Data consistency
⢠Entity deďŹned persistence
Friday, August 23, 13
55. Entity First
⢠Thus far: Operation-Oriented Services:
DoUpdateMatchReport, QueryMatchReport,
FinalizeMatchReport, etc.
⢠Resource-Oriented, the Way of the Web
⢠RESTful resources can be viewed as First Class
Citizens (everything is a resource)
⢠REST method semantics enable use of Cache
Friday, August 23, 13
56. REST in 1 Slide
⢠A REST architecture relies fully on HTTP as an
Application Protocol
⢠REST revolves around URI resources (nouns) and
HTTP methods (actions) and ...
⢠Rely on hypermedia to drive the client(!) state
machine deďŹned by the application
Friday, August 23, 13
57. RESTFul Navajo
⢠Navajo Entity == Resource
⢠Extend to support inheritance and referencing
other Navajo Entities
⢠Add key properties
⢠Navajo Entity support one or more HTTP methods
(GET, PUT, POST and DELETE)
⢠Entity Operations (HTTP methods) are
implemented by Navajo Services
Friday, August 23, 13
58. Entity Service<validations>
<check code=âA MatchReport should have a date unless it is a DRAFT
MatchReportâ>
[/Match/Status] != âDRAFTâ OR CheckDate([/MatchReport/Date)</check>
</validations>
<message name=âMatchReportâ extends="navajo://Report">
<property name="_id" type="string" key="true,auto"/>
<property name="MatchId" type="string" key="true" reference=ânavajo://
Match/MatchIdâ/>
<property name="Status" type="string"/>
<property name="HomeTeamId" type="integer" reference=ânavajo://Team/
TeamIdâ/>
<property name="AwayTeamId" type="integer" reference=ânavajo://Team/
TeamIdâ/>
<message name=âHomePlayersâ type=âarrayâ>
<message name=â HomePlayersâ type=âdeďŹnitionâ>
<property name=âPersonIdâ type=âstringâ reference=ânavajo://
Personâ>
<property name=âRoleâ type=âstringâ/>
</message>
</message>
</message>
Friday, August 23, 13
63. Caching and Consistency
⢠ETag can help us with
⢠Preventing that multiple GET operation
return same data (send 304 instead)
⢠Preventing that Stale data is used for
updating an Entity (check ETag on
PUT,DELETE,POST) (send 409)
Friday, August 23, 13
64. Polyglot Persistence
⢠Entity Operations can implement different
persistence methods
⢠Some Entities can âliveâ inside MongoDB while other
reside in e.g. Oracle
⢠Non-functional data requirements (e.g. consistency,
availability) can drive persistence decision
⢠âTransparentâ migration of Data to other datasources
Friday, August 23, 13
65. Entity drives Service drives Entity drives Service drives Entity
drives Service drives Entity drives Service drives Entity drives
Service
⢠Role of Normal Service and Entity Service
are reversed
⢠Instead of QueryMatchForm,
DoInsertMatchForm,
DoDeleteMatchForm, we have MatchForm
(GET, DELETE, INSERT)
Friday, August 23, 13