Python asyncio REST Resource API

1. GUILLOTINA
An Async REST Resource DB to manage millions of objects

2. CoFounder/CTO Onna - SF/BCN
Connect and search all knowledge
inside an enterprise with ML
Ramon Navarro Bosch
Plone Software Foundation & FWT Member
Web Framework Engineer 😱 !!

3. BACKGROUND

4. “Organizing data is where we spend more time and its boring.”
Data Scientist Sysadmin / Engineers

5. Tables / CSV Docs / Unstructured data
NOSQLSQL

6. WEB FRAMEWORKS
• Angular/React : Server rendering frameworks are
dead
• Most sources of data comes from the web/api
• Lots of experience on storing, distributing,
managing resources

7. Web Framework
Communities
Data Scientist
Communities
Connect live data web framework with data scientist framework

8. Long time ago … 18 years … Zope and ZODB was created
object oriented DB and web application server
Then 16 years ago … Plone was created
layer on top of Z stack to provide CMS
Then 7 years ago … Pyramid was created
merge pylons + repoze.bfg (zope fork)
Then 2 years ago … Plone REST API was created
Abstraction layer for creating resources on top of Plone 5
300 python packages
Then 1 years ago … plone.server was created
Rewrite from scratch of minimum Plone backend with py 3.6 and asyncio

10. TRANSACTION
All operations are managed to
be durable and confirmed,
conflict resolution policies

11. TREE
Information is
organized in
trees of objects

12. RESOURCES
Objects are resources with
schema attributes, annotations,
OO inheritance and static/
dynamic behaviors.
RESOURCE

13. SCHEMA
JSON/
PYTHON
Direct mapping of JSON
schemas and python schemas

14. SECURITY
Full definition of permissions /
roles / principals with global
and local inheritance of
permissions on the tree.Allow,
Deny, Unset,AllowSingle (no
inheritance)

15. CRUD
DynamicTraversal CRUD
HTTP verbs mapping for each
content type
Custom endpoints for specific
operations
GET
HEAD
POST
PUTPATCH

16. ASYNCIO
All based on asyncio for
network integrations with
external indexers, db, services
Based aioHTTP

17. CORS
Cors configured globally and
enabled by default

18. WEBSOCKET
Websocket connection to
apply operations throw
frames. Mapping of REST API
on aTCP async channel.

19. TUS
Binary resumable file upload

20. EVENT
Event based system to trigger
operations in code

21. REGISTRY
Configuration registry x main
container

22. QUEUE
Operational queue and after
response tasks

23. MULTI DB
Mount multiple DBs and
partition objects pickles based
onTree position
POSTGRESQL
COCKROACH

24. FILE CLOUD
Support for S3/GCloud
storage.
***local distributed FS
soon***

25. INDEX
Elasticsearch indexing

26. DIST CACHE
Redis backend

27. SWAGGER
Automatic API documentation
generation

28. CONTAINERS
Docker / K8s / Nomad out
the box

29. EXPLICIT PY
All configuration is defined on
the code using decorators
@configure.service()

30. guillotina(x) = argmin Zope/Plone(x)
* zope.interface still used

31. Traversal
/DB/MAIN_CONTAINER/OBJ1/OBJ2

32. • Only >= Python 3.5
• Designed to host millions of objects
• Memory optimizations
• Apply operations to contained objects in async
• Authentication & authorization extensions (oauth2
flow supported)
• Reusable UI JS components from Plone (Widgets/
SPA)

33. TRY IT !

34. {
"databases": [{
"db": {
"storage": "postgresql",
"transaction_strategy": "resolve",
"dsn": {
"scheme": "postgres",
"dbname": "guillotina",
"user": "postgres",
"host": "localhost",
"password": "",
"port": 5432
},
"pool_size": 40,
"read_only": false
}
}],
"host": "127.0.0.1",
"port": 8080,
"static": [
{"favicon.ico": "static/favicon.ico"}
],
"root_user": {
"password": "root"
},
"cors": {
"allow_origin": ["*"],
"allow_methods": ["GET", "POST", "DELETE", "HEAD", "PATCH"],
"allow_headers": ["*"],
"expose_headers": ["*"],
"allow_credentials": true,
"max_age": 3660
},
"utilities": []
}

35. docker run -p 5432:5432 -d postgres
psql -h localhost -U postgres << EOF
CREATE DATABASE guillotina;
EOF
pip install guillotina
guillotina -c config.json
curl -X GET http://localhost:8080
curl -u root:root http://localhost:8080
curl -X POST -u root:root -H "Content-Type: application/json"
-d '{"@type":"Container","id":"mycontainer","title":"My Lovely Container"}'
http://localhost:8080/db/ | jq .
curl -X POST -u root:root -H "Content-Type: application/json"
-d '{"@type":"Folder","id":"myfolder"}' http://localhost:8080/db/mycontainer | jq .
curl -X POST -u root:root -H "Content-Type: application/json"
-d '{"@type":"Folder"}' http://localhost:8080/db/mycontainer | jq .
curl -X POST -u root:root -H "Content-Type: application/json"
-d '{"@type":"Item","id":"myitem"}' http://localhost:8080/db/mycontainer/myfolder | jq .
curl -X PATCH -u root:root -H "Content-Type: application/json"
-d '{"title": "My new title"}' http://localhost:8080/db/mycontainer/myfolder/myitem | jq .
curl -X GET -u root:root -H "Content-Type: application/json"
-d '{"title": "My new title"}' http://localhost:8080/db/mycontainer/myfolder/myitem | jq .
curl -X DELETE -u root:root -H "Content-Type: application/json"
http://localhost:8080/db/mycontainer | jq .

36. DATA MODEL
Resource &
Container
Interface
Schema fields
Static
Behaviors
Dynamic
Behaviors

37. from guillotina import configure
from guillotina.content import Item
from guillotina.interfaces import IItem
from guillotina import schema
class ICustomType(IItem):
foo = schema.Text()
@configure.contenttype(
type_name="CustomType",
schema=ICustomType,
behaviors=[
"guillotina.behaviors.dublincore.IDublinCore",
"example.behaviors.ICustomBehavior",
])
class CustomType(Item):
pass

38. @configure.subscriber(for_=(ICustomType, IObjectAddedEvent))
async def created_userfolder(obj, evnt):
...
@configure.service(
context=ICustomType, name='@myEndpoint', method='GET',
permission='guillotina.AccessContent')
class MyEndpoint(Service):
async def __call__(self):
...

39. WIP

40. WIP : DISTRIBUTED HIVE
Execute an operation to all objects in distributed execution
Based on etcd
Dynamic workers that are going to compute a task
No aggregation callback
Batch mass modification of the model
guillotina_hive
(thanks @vangheezy)

41. from guillotina.traversal import traverse
async def my_task(task_info, root, request):
data = task_info.data
path = data['path']
ob, end_path = await traverse(request, root, path.lstrip('/').split('/'))
assert len(end_path) == 0
from guillotina.component import getUtility
from guillotina_hive.interfaces import IHiveUtility
hive = getUtility(IHiveUtility)
task_info = TaskInfo('my_task', {'foo': 'bar'})
await hive.push_task(task_info)

42. WIP : INFERENCE
Right now using tf serving
Storing models as resources on the api
TF loader adapter to get the model with cache
Manages tf serving service provisioning on k8s
guillotina_tf
URI_RESOURCE/@applyModel?model=URI_MODEL
Use event system on create/update apply model

43. WIP : DISTRIBUTEDTRAINING
Right now using tf distributed
Storing models as resources on the api
Start workers and parameters servers with k8s
Offer the tf operation to add guillotina REST API as source
Reinforcement learning support
ML architecture definition on guillotina with RestrictedPython
guillotina_tflearn
send_to_learn(FEED,ARCHITECTURE, MODEL_URL)
Websocket feedback

44. “Why gRPC has no asyncio support?”

45. “Made in Barcelona, looking for contributions”

46. l
October 16 22 2017
Come to the Digital Experience Conference
2017.ploneconf.org

47. PREGUNTES ?
GRÀCIES !
ramon@plone.org
CORE : https://github.com/plone/guillotina
DOCS : http://guillotina.readthedocs.io/en/latest/
PYPI : https://pypi.python.org/pypi/guillotina
MODULES : https://github.com/guillotinaweb