SlideShare ist ein Scribd-Unternehmen logo
1 von 31
Downloaden Sie, um offline zu lesen
Content Delivery
At Aviary
MongoDb User Group 11/19/13
Aviary
Photo-Editing
SDK & Apps
Fully-Baked UI
Configurable, High-Quality Tools
Over 5,000 Partners
Over 50 Million Monthly Users
Over 4 Billion Photos Edited
iOS, Android, Web, Windows, Server
J
Who Are We?
Nir

Jack

Lead Serverside

Director of

Engineer

Engineering

Likes:
●
●
●

Automated deployment
Big-O notation
Brainteasers

Hates:
●

Cilantro

Likes:
●
●
●

Parallelizing processes
DRY code
Seltzer

Hates:
●

Food after the sell-by date
Content
Effects

Frames

Stickers

Messages

J
The “Fat Tiny” Situation
We want to dynamically deliver the Fat Tiny
stickers to our users

How can we do that?
J
The CDS
Aviary’s Content Delivery System
Version 1
● Static files on S3
● Excess data sent
● Data stored in MySQL
● New features meant new code

N
The “Fat Tiny” Situation
We want to display the Fat Tiny sticker pack only
to Picstitch users in Japan who use iOS 7

How can we do that?
N
CDS V2
Version 2 Overview
Stack
Global
Users

Akamai

AWS Cloud Formation
AWS Elastic LoadBalancer (ELB)
Node.js
AWS Elastic Cloud Computing (EC2)
Ubuntu
Bash
MongoDb
JSON
JSON Schema

Content
Delivery
Network

Load Balancer

API
Servers
Database
Cluster
Management
System

N
Why MongoDB?
● Great with Node (JSON-based)
● Schema-less is easy to change/query
● Read-heavy
● Relatively small data set

N
How It Works
A Behind-the-Scenes Look
Delivered Types
● Manifest JSON
○ Content Set
○ Content Versions

● Content JSON
○ Content Metadata
○ All Assets

N
The “Fat Tiny” Situation
We want to manage the Fat Tiny sticker pack
as a single entity, but we want to deliver it to
each device in its own
optimal format.
How can we do that?
N
Response Formatting Model
Content Entry

Response Formats

Responses

JSON document describing
content item

JSON documents defining mappings
from entry to responses

Actual JSON responses
delivered to devices

J
Content Formats
● JSON schema

{
"type":"object",
"properties":{
"metadata": {

● Added properties

"type": "object",
"properties":{
"displayName": {
"type": "string",

● Custom types

}
}
},

● Validates entries

"icon": {
"type": "object",
"customType": "image"
"properties":{
"path": {
"type": "string",
"required": true,

J
Response Formats
● JSON schema

{
"type":"object",
"properties":{
"identifier": {

● dataKey property

"type": "string",
"dataKey": "identifier"
},

● Defines response

"name": {
"type": "string",
"dataKey": "metadata.displayName"
},

structure

"iconImagePath": {
"type": "string",
"dataKey": "icon.path-100"

● Maps content

},
"items": {
"type": "array",
"dataKey": "items"

J
Content Deployment
1. Insert/Update CMS Entry
"identifier": "com.aviary.stickers.234fe"

2. Find Response Formats
"type":"object",

"id": "com.aviary.stickers.234fe",

"properties":{

"metadata": {
"displayName": "Hats"

"name": "Hats",

"id": {
"type": "string",

},

"dataKey": "identifier"

"icon": {
"path": "cds/hats/icon.png"

"iconImagePath": "cds/hats/icon100.png"
"stickers": [
{

},

"identifier": "1"

"name": {

"path-100": "cds/hats/icon100.png"

"type": "string",

},

"imageUrl": "cds/hats/1.png"

"dataKey": "metadata.displayName"

"items": [

},

{

"type": "string",

"imageUrl": "cds/hats/1.png"
}

}
],

"iconImagePath": {

"identifier": "1"

]

3. Generate+Insert Responses

"versionKey": "e4532fd342"

"dataKey": "icon.path-100"
},
"stickers": {
"type": "array",
"dataKey": "items"

J
Manifests
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
}
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]

J
Manifest Deployment
Using Aggregate to Find the Newest Versions

Manifest with Correct Version Keys

{
$match:{

"stickers": [
{

formatId:{$in:formatIds},

"id": "com.aviary.stickers.234fe",

identifier:{$in:identifiers}

"versionKey": "e4532fd342"

}
}

},
],

{
$sort: { _id: -1 }

"effects": [
{

},

"id": "com.aviary.effects.25435",

{

"versionKey": "fd4324323"

$group: {
}

_id: "$identifier",
versionKey:{$first:"$versionKey"}

]

}
}

J
Scopes and Targeting
Deployed Manifests Have Scopes

End Users Have Scope Parameters

Manifest 1
"targetingScope": {
"apiKey": "abc",
"country": ["JP"]
},

/manifest?

"formattingScope": {
"platform": "ios",
"minOsVersion": "7.0.0"
}

apiKey=abc&
country=JP&
language=ja&
platform=ios&

Manifest 2

osVersion=7.2.0

"targetingScope": {
"apiKey": "def",
},
"formattingScope": {
"platform": "android",
"minOsVersion": "6.0.0"
}

N
API Servers
Scope parameters are converted into queries
Manifest 1
"targetingScope": {
db.manifest.find({

"apiKey": "abc",

"apiKey": {$in: ["abc", null]},
/manifest?

country=JP&

"formattingScope": {

"platform": {$in: ["ios", null]},
7002000

"platform": "ios",

"minOsVersion": {$lte: 7002000}

language=ja&

osVersion=7.2.0

},

"language": {$in: ["ja", null]},

apiKey=abc&

platform=ios&

"country": ["JP"]

"country": {$in: ["JP", null]},

"minOsVersion": "7.0.0"

}).sort({

}

"apiKey": -1,

Manifest 2

"language": -1,

"targetingScope": {

"country": -1,

"apiKey": "abc",

"minOsVersion": -1,

},

"platform": -1,

"formattingScope": {

"_id": -1

"platform": "ios",

}).limit(1)

"minOsVersion": "6.0.0"
}

N
Versioned Content
Received Manifests Contain VersionKeys
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"

db.content.findOne({
/content?
versionKey=e4532fd342

}

"versionKey": “e4532fd342”
});

],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]

N
Response Caching
db.manifests.find({

/manifest?

"apiKey": {$in: ["abc", null]},

apiKey=abc&

"country": {$in: ["JP", null]},

country=JP&

"language": {$in: ["ja", null]},

language=ja&

"platform": {$in: ["ios", null]},

platform=ios&

"minOsVersion": {$gte: 7002000}

osVersion=7.2.0

}).sort({
…,
"_id": -1
}).limit(1)

db.cachedManifests.findOne({
"url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios"
})

N
PAULA
The CDS Management Console
Auto-generated UI

J
Other Mongo Usage
● PAULA permissions in user objects
users collection
{
"name": "nir",
"email": "nir@aviary.com",
"permissions": [
"content",
"dev",
"admin",
"partying"
]
}

● Integration tests interact with schemaless db willy nilly
N
Conclusion
The Takeaway
The Facts
● Built and deployed in 3 months
● Very few struggles with MongoDB
● Seamless management
● Graceful scaling from 0 to over 20M MAUs
● Happy serverside engineers
J
The Future
● Targeted Translations
● Granular User Targeting
● PAULA for the masses

N
Questions?
Comments also welcome
nir@aviary.com

jack@aviary.com

…and by the way, WE’RE HIRING!

Weitere ähnliche Inhalte

Ähnlich wie MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson

RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)contest-theta360
 
Engineering the New LinkedIn Profile
Engineering the New LinkedIn ProfileEngineering the New LinkedIn Profile
Engineering the New LinkedIn ProfileJosh Clemm
 
Exploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptExploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptwesley chun
 
Build 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive CardsBuild 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive CardsWindows Developer
 
Getting started with titanium
Getting started with titaniumGetting started with titanium
Getting started with titaniumNaga Harish M
 
Using Google (Cloud) APIs
Using Google (Cloud) APIsUsing Google (Cloud) APIs
Using Google (Cloud) APIswesley chun
 
Architecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystemArchitecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystemYael Garten
 
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemStrata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemShirshanka Das
 
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...Frédéric Harper
 
Getting started with Appcelerator Titanium
Getting started with Appcelerator TitaniumGetting started with Appcelerator Titanium
Getting started with Appcelerator TitaniumTechday7
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Tech Community
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Tech Community
 
Building Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchBuilding Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchMongoDB
 
iPhone/iPad Development with Titanium
iPhone/iPad Development with TitaniumiPhone/iPad Development with Titanium
iPhone/iPad Development with TitaniumAxway Appcelerator
 
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CAAppcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CAJeff Haynie
 
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morePower your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morewesley chun
 
Fixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blockingFixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blockingIgalia
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점Jeado Ko
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점 Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점 WebFrameworks
 

Ähnlich wie MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson (20)

RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
 
Engineering the New LinkedIn Profile
Engineering the New LinkedIn ProfileEngineering the New LinkedIn Profile
Engineering the New LinkedIn Profile
 
Exploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptExploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScript
 
Build 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive CardsBuild 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive Cards
 
Getting started with titanium
Getting started with titaniumGetting started with titanium
Getting started with titanium
 
Using Google (Cloud) APIs
Using Google (Cloud) APIsUsing Google (Cloud) APIs
Using Google (Cloud) APIs
 
The Rise of NoSQL
The Rise of NoSQLThe Rise of NoSQL
The Rise of NoSQL
 
Architecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystemArchitecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystem
 
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemStrata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
 
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
 
Getting started with Appcelerator Titanium
Getting started with Appcelerator TitaniumGetting started with Appcelerator Titanium
Getting started with Appcelerator Titanium
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needs
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needs
 
Building Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchBuilding Your First App with MongoDB Stitch
Building Your First App with MongoDB Stitch
 
iPhone/iPad Development with Titanium
iPhone/iPad Development with TitaniumiPhone/iPad Development with Titanium
iPhone/iPad Development with Titanium
 
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CAAppcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CA
 
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morePower your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
 
Fixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blockingFixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blocking
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점 Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
 

Mehr von Hakka Labs

Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)Hakka Labs
 
DataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series searchDataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series searchHakka Labs
 
DataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data ScienceDataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data ScienceHakka Labs
 
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataDatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataHakka Labs
 
DataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at InstacartDataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at InstacartHakka Labs
 
DataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scaleDataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scaleHakka Labs
 
DataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor DataDataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor DataHakka Labs
 
DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale Hakka Labs
 
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQDataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQHakka Labs
 
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...Hakka Labs
 
DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...Hakka Labs
 
DataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at PinterestDataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at PinterestHakka Labs
 
DataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineeringDataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineeringHakka Labs
 
DataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data StructuresDataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data StructuresHakka Labs
 
DataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using SparkDataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using SparkHakka Labs
 
DataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with OurselvesDataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with OurselvesHakka Labs
 
DataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High DeliverabilityDataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High DeliverabilityHakka Labs
 
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...Hakka Labs
 
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedInDataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedInHakka Labs
 
DataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL WorkshopDataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL WorkshopHakka Labs
 

Mehr von Hakka Labs (20)

Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)
 
DataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series searchDataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series search
 
DataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data ScienceDataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data Science
 
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataDatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
 
DataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at InstacartDataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at Instacart
 
DataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scaleDataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scale
 
DataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor DataDataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
 
DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale
 
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQDataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
 
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
 
DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...
 
DataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at PinterestDataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at Pinterest
 
DataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineeringDataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineering
 
DataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data StructuresDataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data Structures
 
DataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using SparkDataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
 
DataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with OurselvesDataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with Ourselves
 
DataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High DeliverabilityDataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
 
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
 
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedInDataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
 
DataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL WorkshopDataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL Workshop
 

Kürzlich hochgeladen

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfSeasiaInfotech2
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 

Kürzlich hochgeladen (20)

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdf
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 

MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson

  • 2. Aviary Photo-Editing SDK & Apps Fully-Baked UI Configurable, High-Quality Tools Over 5,000 Partners Over 50 Million Monthly Users Over 4 Billion Photos Edited iOS, Android, Web, Windows, Server J
  • 3. Who Are We? Nir Jack Lead Serverside Director of Engineer Engineering Likes: ● ● ● Automated deployment Big-O notation Brainteasers Hates: ● Cilantro Likes: ● ● ● Parallelizing processes DRY code Seltzer Hates: ● Food after the sell-by date
  • 5. The “Fat Tiny” Situation We want to dynamically deliver the Fat Tiny stickers to our users How can we do that? J
  • 6. The CDS Aviary’s Content Delivery System
  • 7. Version 1 ● Static files on S3 ● Excess data sent ● Data stored in MySQL ● New features meant new code N
  • 8. The “Fat Tiny” Situation We want to display the Fat Tiny sticker pack only to Picstitch users in Japan who use iOS 7 How can we do that? N
  • 9. CDS V2 Version 2 Overview
  • 10. Stack Global Users Akamai AWS Cloud Formation AWS Elastic LoadBalancer (ELB) Node.js AWS Elastic Cloud Computing (EC2) Ubuntu Bash MongoDb JSON JSON Schema Content Delivery Network Load Balancer API Servers Database Cluster Management System N
  • 11. Why MongoDB? ● Great with Node (JSON-based) ● Schema-less is easy to change/query ● Read-heavy ● Relatively small data set N
  • 12. How It Works A Behind-the-Scenes Look
  • 13. Delivered Types ● Manifest JSON ○ Content Set ○ Content Versions ● Content JSON ○ Content Metadata ○ All Assets N
  • 14. The “Fat Tiny” Situation We want to manage the Fat Tiny sticker pack as a single entity, but we want to deliver it to each device in its own optimal format. How can we do that? N
  • 15. Response Formatting Model Content Entry Response Formats Responses JSON document describing content item JSON documents defining mappings from entry to responses Actual JSON responses delivered to devices J
  • 16. Content Formats ● JSON schema { "type":"object", "properties":{ "metadata": { ● Added properties "type": "object", "properties":{ "displayName": { "type": "string", ● Custom types } } }, ● Validates entries "icon": { "type": "object", "customType": "image" "properties":{ "path": { "type": "string", "required": true, J
  • 17. Response Formats ● JSON schema { "type":"object", "properties":{ "identifier": { ● dataKey property "type": "string", "dataKey": "identifier" }, ● Defines response "name": { "type": "string", "dataKey": "metadata.displayName" }, structure "iconImagePath": { "type": "string", "dataKey": "icon.path-100" ● Maps content }, "items": { "type": "array", "dataKey": "items" J
  • 18. Content Deployment 1. Insert/Update CMS Entry "identifier": "com.aviary.stickers.234fe" 2. Find Response Formats "type":"object", "id": "com.aviary.stickers.234fe", "properties":{ "metadata": { "displayName": "Hats" "name": "Hats", "id": { "type": "string", }, "dataKey": "identifier" "icon": { "path": "cds/hats/icon.png" "iconImagePath": "cds/hats/icon100.png" "stickers": [ { }, "identifier": "1" "name": { "path-100": "cds/hats/icon100.png" "type": "string", }, "imageUrl": "cds/hats/1.png" "dataKey": "metadata.displayName" "items": [ }, { "type": "string", "imageUrl": "cds/hats/1.png" } } ], "iconImagePath": { "identifier": "1" ] 3. Generate+Insert Responses "versionKey": "e4532fd342" "dataKey": "icon.path-100" }, "stickers": { "type": "array", "dataKey": "items" J
  • 19. Manifests "stickers": [ { "id": "com.aviary.stickers.234fe", "versionKey": "e4532fd342" }, { "id": "com.aviary.stickers.fed34", "versionKey": "c54532343d" } ], "frames": [ { "id": "com.aviary.frames.25435", "versionKey": "fd4324323" } ] J
  • 20. Manifest Deployment Using Aggregate to Find the Newest Versions Manifest with Correct Version Keys { $match:{ "stickers": [ { formatId:{$in:formatIds}, "id": "com.aviary.stickers.234fe", identifier:{$in:identifiers} "versionKey": "e4532fd342" } } }, ], { $sort: { _id: -1 } "effects": [ { }, "id": "com.aviary.effects.25435", { "versionKey": "fd4324323" $group: { } _id: "$identifier", versionKey:{$first:"$versionKey"} ] } } J
  • 21. Scopes and Targeting Deployed Manifests Have Scopes End Users Have Scope Parameters Manifest 1 "targetingScope": { "apiKey": "abc", "country": ["JP"] }, /manifest? "formattingScope": { "platform": "ios", "minOsVersion": "7.0.0" } apiKey=abc& country=JP& language=ja& platform=ios& Manifest 2 osVersion=7.2.0 "targetingScope": { "apiKey": "def", }, "formattingScope": { "platform": "android", "minOsVersion": "6.0.0" } N
  • 22. API Servers Scope parameters are converted into queries Manifest 1 "targetingScope": { db.manifest.find({ "apiKey": "abc", "apiKey": {$in: ["abc", null]}, /manifest? country=JP& "formattingScope": { "platform": {$in: ["ios", null]}, 7002000 "platform": "ios", "minOsVersion": {$lte: 7002000} language=ja& osVersion=7.2.0 }, "language": {$in: ["ja", null]}, apiKey=abc& platform=ios& "country": ["JP"] "country": {$in: ["JP", null]}, "minOsVersion": "7.0.0" }).sort({ } "apiKey": -1, Manifest 2 "language": -1, "targetingScope": { "country": -1, "apiKey": "abc", "minOsVersion": -1, }, "platform": -1, "formattingScope": { "_id": -1 "platform": "ios", }).limit(1) "minOsVersion": "6.0.0" } N
  • 23. Versioned Content Received Manifests Contain VersionKeys "stickers": [ { "id": "com.aviary.stickers.234fe", "versionKey": "e4532fd342" }, { "id": "com.aviary.stickers.fed34", "versionKey": "c54532343d" db.content.findOne({ /content? versionKey=e4532fd342 } "versionKey": “e4532fd342” }); ], "frames": [ { "id": "com.aviary.frames.25435", "versionKey": "fd4324323" } ] N
  • 24. Response Caching db.manifests.find({ /manifest? "apiKey": {$in: ["abc", null]}, apiKey=abc& "country": {$in: ["JP", null]}, country=JP& "language": {$in: ["ja", null]}, language=ja& "platform": {$in: ["ios", null]}, platform=ios& "minOsVersion": {$gte: 7002000} osVersion=7.2.0 }).sort({ …, "_id": -1 }).limit(1) db.cachedManifests.findOne({ "url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios" }) N
  • 27. Other Mongo Usage ● PAULA permissions in user objects users collection { "name": "nir", "email": "nir@aviary.com", "permissions": [ "content", "dev", "admin", "partying" ] } ● Integration tests interact with schemaless db willy nilly N
  • 29. The Facts ● Built and deployed in 3 months ● Very few struggles with MongoDB ● Seamless management ● Graceful scaling from 0 to over 20M MAUs ● Happy serverside engineers J
  • 30. The Future ● Targeted Translations ● Granular User Targeting ● PAULA for the masses N