This document discusses data product architectures and provides examples of different architectures for data products, including the lambda architecture, analyst architecture, recommender architecture, and partisan discourse architecture. It also discusses common design principles for data product architectures, such as using microservices with stateful backend services and database-backed APIs. Key aspects of data product architectures include handling training data and models, making predictions via APIs, updating models and annotations, and designing flexible systems that can incorporate new models and data.

1. Data Product
Architectures
Benjamin Bengfort
@bbengfort
District Data Labs

2. Abstract

3. What is data science?
Or what is the goal of data science?
Or why do they pay us so much?

4. Two Objectives Orient Data Science to Users

5. Data Products are self-adapting, broadly
applicable software-based engines that
derive their value from data and generate
more data by influencing human behavior
or by making inferences or predictions
upon new data.

6. Data Products are Applications that
Employ Many Machine Learning Models

7. Data Report

8. Without Feedback
Models are Disconnected
They cannot adapt, tune, or react.

11. Data Products aren’t
single models
So how do we architect data products?

12. The Lambda Architecture

14. Three Case Studies

15. Analyst Architecture

16. Analyst Architecture: Document Review

17. Analyst Architecture: Triggers

18. Recommender Architecture

19. Recommender: Annotation Service

20. Partisan Discourse Architecture

21. Partisan Discourse: Adding Documents

22. Partisan Discourse: Documents

23. Partisan Discourse: User Specific Models

24. Commonalities?

26. Microservices Architecture: Smart Endpoints, Dumb Pipe
HTTP
HTTP
HTTP
HTTP
HTTPHTTP
HTTP
Stateful
Services
Database Backed
Services

28. Django Application Model

29. Class Based, Definitional Programming
from rest_framework import viewsets
class InstanceViewSet(viewsets.ModelViewSet):
queryset = Instance.objects.all()
serializer_class = InstanceSerializer
def list(self, request):
pass
def create(self, request):
pass
def retrieve(self, request, pk=None):
pass
def update(self, request, pk=None):
pass
def destroy(self, request, pk=None):
pass
from django.db import models
from rest_framework import serializers as rf
class InstanceSerializer(rf.ModelSerializer):
prediction = rf.CharField(read_only=True)
class Meta:
model = Instance
fields = ('color', 'shape', 'amount')
class Instance(models.Model):
SHAPES = ('square', 'triangle', 'circle')
color = models.CharField(default='red')
shape = models.CharField(choices=SHAPES)
amount = models.IntegerField()

30. Features and Instances as Star Schema

31. REST API Feature Interaction

32. Model (ML) Build Process: Export Instance Table
COPY (
SELECT instances.* FROM instances
JOIN feature on feature.id = instance.id
...
ORDER BY instance.created
LIMIT 10000
) as instances
TO '/tmp/instances.csv' DELIMITER ',' CSV HEADER;

33. Model (ML) Build Process: Build Model
import pandas as pd
from sklearn.svm import SVC
from sklearn.cross_validation import KFold
# Load Data
data = pd.read_csv('/tmp/instances.csv')
scores = []
# Evaluation
folds = KFold(n=len(data), n_folds=12)
for train, test in folds:
model = SVC()
model.fit(data[train])
score = model.score(data[test])
scores.append(score)
# Build the actual model
model = SVC()
model.fit(data)

34. Model (ML) Build Process: Store Model
import json
import pickle
import base64
import datetime
data = pickle.dump(model)
data = base64.base64encode(data)
return {
"model": data,
"created": datetime.datetime.now(),
"form": repr(model),
"name": model.__class__.__name__,
"scores": scores,
}

35. Model Data Storage
from django.db import models
class PredictiveModel(models.Model):
name = models.CharField()
params = models.JSONField()
build = models.FloatField()
f1_score = models.FloatField()
created = models.DateTimeField()
data = models.BinaryField()

36. REST API Model Interaction
featurize()
predict()
Models Stored in
Memory
Update
Annotations

37. Build Data Products!

39. Questions?