I Love APIs 2015

1. 1
Apache Usergrid: Web Scale Mobile APIs
with ElasticSearch and Cassandra!
Todd Nine
Principal Architect
Jeffrey West
BaaS Engineering
Coach

2. Agenda
2
1.
Why do we need Usergrid?
2.
What is Usergrid & How does it work?
3.
How do we achieve web scale?
4.
Learnings
5.
Roadmap & Questions
©2015 Apigee. All Rights Reserved.

3. 3
(m) obile
B ackend
a s
a
S ervice
Datastore
Push
Notifications
Connections /
Social
Location
Users
…and more
Application

4. Systems of Engagement
4

5. Building Systems of Engagement with Edge API BaaS
5
Systems of "
Record
Apigee Edge: Intelligent API Platform
Systems of
Engagement
App Developers
Database
ERP
CRM
Data "
Warehouse
Data Lake
Partner Apps
Consumer Apps
Employee Apps
Insights

6. 6
©2015 Apigee. All Rights Reserved.
Web Scale"
10’s of thousands requests/sec"
…across multiple data centers

7. 7
©2015 Apigee. All Rights Reserved.
Web Scale / MongoDB

8. Usergrid
8

9. • Apache Open Source Mobile Backend as a Service
• REST API for easy interaction from any platform
• SDKs available for iOS / Android / Javascript
9

10. The Usergrid Taxonomy
10

11. Organization
Organization
Organization
Application
Devices
Users
Groups
Security
Roles
Collection
Collection
Entity
{…}
Entity
{…}
Entity
{…}
Application
Devices
Users
Groups
Security
Roles
Entity
{…}
Entity
{…}
Entity
{…}
Collection
Collection
Entity
{…}
Entity
{…}
Entity
{…}

12. But I just want persistence!
12
©2015 Apigee. All Rights Reserved.
REST API
GET / PUT / POST / DELETE
Scalable, Multi-tenant
Persistence (Single Cluster)

13. What people do seeking ‘web scale’?
13
©2015 Apigee. All Rights Reserved.

14. The Usergrid Stack
14

15. • Canonical System of Record
• Collections
– Unique Values / Primary Keys
• Entities
– Schemaless design
– O(1) entity retrieval
• Connections / Graph
– Scalable Relationships
Horizontally Scalable Persistence

16. Built on Horizontally Scalable Technologies
• Filter / Limit / Cursor:
– GET http://api.usergrid.com/{org}/{app}/{collection}?
ql=select * where title = 'A Tale of Two
Cities’&limit=50&cursor=LTIxNDg0NDUxNDpnR2tBQVFFQWdITUFDW
FJ2YlM1emJXbDBhQUNBZFFBUUQyMVZneExfRWVLRlV3TG9Hc1doZXdDQW
RRQVFIYVdjb0JwREVlS1VCd0xvR3NWT0JRQQ
– GET http://api.usergrid.com/{org}/{app}/{collection}?
ql=select * where location within {d} of {x, y}
17
©2015 Apigee. All Rights Reserved.
( )

17. Usergrid Application
18
©2015 Apigee. All Rights Reserved.

18. Architecture: Multi-Region Indexing & Consistency
SQS
SQS
SNS
6) INDEX
6) INDEX
Region B
Region A
C* Replication
2)WRITE
5)READ
5)READ

19. Web Scale
20

20. Bottlenecks
21
©2015 Apigee. All Rights Reserved.
Tomcat
20% CPU
!?? WHY ??!
Cassandra
10% CPU

21. Iterative Process
22
©2015 Apigee. All Rights Reserved.
10,266/s
9,700/s
7,233/s
3,750

22. Hurdles
• Gatling
– Can’t get 10k/s with one node
• Cassandra
– CPU
• Compression?
• C3.2xlarge -> c3.4xlarge
• Row caching?
• Tomcat
– HTTP Connector
• acceptCount
• maxConnections
• maxThreads
• Non-blocking IO?
• Usergrid
– # of reads to Cassandra
23
©2015 Apigee. All Rights Reserved.

23. Scaling Usergrid to 10,000 Req/s w/ Gatling
• 10k test run:
– 35 c3.xlarge Tomcat Nodes
– 9 c3.4xlarge C* nodes
– 6 m3.large ES nodes
• Closing test ~7,800/s:
– 15 c3.xlarge Tomcat Nodes
– 9 c3.4xlarge C* nodes
– 6 m3.large ES nodes
24

24. Elasticsearch
(working… not so well…)
25

25. Learn from our mistakes
26

26. Risks of Dynamic Mapping
27
©2015 Apigee. All Rights Reserved.
D
D
D
D
M

27. Risks of Dynamic Mapping
28
©2015 Apigee. All Rights Reserved.
DD
D
DD
DD
D
DD
D D
DD
DD
DD
DD
Inbound message
requires a mapping
update
Mapping updated &
broadcast to the
cluster
M

28. Risks of Dynamic Mapping
29
©2015 Apigee. All Rights Reserved.
M
DD
D
DD
DD
D
DD
D D
DD
DD
DD
DD
At scale (full loaded)
only one mapping
update can
processed at a time

29. Risks of Dynamic Mapping
30
©2015 Apigee. All Rights Reserved.
DD
D
DD
DD
D
DD
D D
DD
DD
DD
DD
M

30. Risks of Dynamic Mapping
31
©2015 Apigee. All Rights Reserved.
M
DD
D
DD
DD
D
DD
D D
DD
DD
DD
DD

31. 30 ES Data Nodes
X
100MB mapping file
=
3,000MB
For each schema/
mapping update
3GB data transfer
X
5M entities with
different schemas
=
a LOT of data
transfer and a lot
of latency
Risks of Dynamic Mapping
32
©2015 Apigee. All Rights Reserved.
DD
D
DD
DD
D
DD
D D
DD
DD
DD
DD
M

32. Elasticsearch
(working well)
33

33. {
"hello": {
"world":{
"message": "Hello!"
},
"items": [
{
"name": "apples",
"weight": 100
}
]
}
}
{
"uuid": "48e262d6-b922-11e4-8a11",
"ud_entity_id": "ad2b33c2e",
"size": 100,
"ug_app": "1a67aed7-bbb3-11e4-8a11",
"fields": [
{
”name": "hello.world.message",
”string": "Hello!"
},
{
”name": "items[0].name",
”string": "apples"
},
{
”name": "items[0].weight",
"integer": 100
}
]
}
Flattened Documents
34
©2015 Apigee. All Rights Reserved.
Indexed as

34. Data Node A
Data Node B
Data Node C
Index Allocation
• Multiple Indexes spread across the cluster
• Indexes have shards
• Read/Write aliases point to physical
indexes
• Aliases assigned based on hashing of the
App UUID
35
©2015 Apigee. All Rights Reserved.
Index_1
1
1
2
3
2
3
app_1_read_alias
app_1_write_alias

35. Data Node A
Data Node B
Data Node C
Tenant Management
36
©2015 Apigee. All Rights Reserved.
Index_1
1
1
2
3
2
3
app_1_read_alias
app_1_write_alias
app_2_read_alias
app_2_write_alias

36. Data Node A
Data Node B
Data Node C
Tenant Management
• Assign Write alias to new Index
• Add new index to read alias
• Stream data from old index
• Delete data from old index
37
©2015 Apigee. All Rights Reserved.
Index_1
1
1
2
3
2
3
Index_2
1
1
2
3
2
3
app_2_read_alias
app_2_write_alias
stream2es
app_1_read_alias
app_1_write_alias

37. Data Node D
Data Node A
Data Node B
Data Node C
Tenant Management
• Indexes targeted to separate machines
38
©2015 Apigee. All Rights Reserved.
Index_1
1
1
2
3
2
3
Index_2
1
1
2
3
2
3
app_1_read_alias
app_1_write_alias
app_2_read_alias
app_2_write_alias

38. Data Node A
Data Node B
Data Node C
Data Node A
Data Node B
Data Node C
Multiple Scaling Vectors
39
©2015 Apigee. All Rights Reserved.
Index_1
1
1
2
3
2
3
Index_2
1
1
2
3
2
3
Index_3
1
1
2
3
2
3
Index_4
1
1
2
3
2
3
Index_5
1
1
2
3
2
3

39. Working with Graph & NoSQL Data
40

40. Data De-normalization per interaction
41
ERP
Product
CatalogsCRM Data Lake
Data
Warehouse

41. Graph Data Persistence: Entities & Connections
• Entities
– Objects / Documents representing an Entity
• Collections
– Logical grouping of Entities
• Connections / Entity Relationships
– Graph Data Structure
– /{collection}/{entityA}/{verb}/{collection}/
{entityB}
– /Users/me/likes/Products/iPhone6
– /Recipes/Bread/includes/Ingredients/Flour
42
User
A
Product
A
User
B
User
C
Likes>>
<<Liking

42. Roadmap
43

43. What’s Next for Usergrid
• Grow the community!
• Blueprints Driver Release!
• Service abstraction
• Indexing closer to real-time
• Larger Clusters
• Distributed Work System
• Encryption & Security
44

44. http://usergrid.apache.org
45