OpenIO presentation about serverless computing and object storage for AI applications during the Paris Open Source Summit 2017

1. Next-Gen Object Storage
and Serverless Computing

2. OpenIO
A practical approach to
Serverless Computing and
Next-gen Object Storage for AI applications
Enrico Signoretti – Head of Product Strategy
OpenIO

3. OpenIO
1PB of Flash per Rack Unit!

4. OpenIO
IoT and Edge Computing

5. OpenIO
Machine Learning and Artificial Intelligence
OpenIO

6. OpenIO
Topics
1
2
3
Evolution of Computing Models
Serverless Computing
OpenIO Grid for Apps

7. OpenIO
General purpose
Traditional Workloads
VMs, DBs, …
Specialized tasks
AI, ML, VR…
Faster Response
Hardware evolution
X86 ARM Coprocessors
Addressing changing needs and workloads
Power efficiency
Edge, IoT, High Density
Massive Parallelization
and Workload distribution

8. OpenIO
Computing models evolution
Virtualization Containers Serverless
From compute to datacentric workloads
Stateless ComputingLegacy Consolidation Event-driven processing
Amazon EC2 Google Container Engine Amazon Lambda over S3
VMware Kubernetes / Docker OpenIO
Time >

9. “
”OpenIO
It is a cloud computing model for which server
management and capacity planning decisions
are completely hidden from the developer or
the operator
What is Serverless Computing?

10. “
”OpenIO
Automate tasks and offload them
to the infrastructure
Why?

11. OpenIO
More $/DataBetter $/GB
Next-Generation
Object Storage
Serverless
Computing

12. OpenIO
Traditional computing
Orchestration layer
Application Application
Containers
Operating System(s)
Hypervisor
Data storage
Application
Data processing triggered by events
Data storage
Serverless computing
OpenIO

13. “
”OpenIO
It simplifies and speeds up
many workloads
Why?

14. OpenIO
Why?
It’s for everyone
(Developers and SysAdmins)

15. OpenIO
Use cases?
You name it!

16. OpenIO
OpenIO Grid for Apps
Let’s start with a demo

17. OpenIO
Grid for Apps in practice
Category tag = Volcano
Queue manager
Everything happens into SDS (no external resources)
OpenIO SDS
Metadata enrichment
Add Category metadata tag to original object
Processing Indexing

18. OpenIO
Grid for Apps in practice
Faces tag = 10
Queue manager
Everything happens into SDS (no external resources)
OpenIO SDS
Metadata enrichment
Add Faces metadata tag to original object
Processing

19. OpenIO
#!/usr/bin/env python
import cv2
import json
import numpy as np
from oio.api import object_storage
from oio.event.beanstalk import Beanstalk, ResponseError
def faceclassifier(image):
# Specify the trained cascade classifier
face_cascade_name = "/etc/oio/sds/OPENIO/haarcascade_frontalface_alt.xml"
# Create a cascade classifier
face_cascade = cv2.CascadeClassifier()
# Load the specified classifier
face_cascade.load(face_cascade_name)
# Run the classifiers
faces = face_cascade.detectMultiScale(image, 1.1, 5, 0|cv2.cv.CV_HAAR_SCALE_IMAGE, (30, 30))
return faces
b = Beanstalk.from_url("beanstalk://127.0.0.1:6014")
b.watch("oio-process")
while True:
try:
event_id, data = b.reserve()
except ResponseError:
continue
# Retrieve the information from the event (namespace, bucket, object name ...)
meta = json.loads(data)
url = meta["url"]
s = object_storage.ObjectStorageAPI(url["ns"], "127.0.0.1:6006")
meta, stream = s.object_fetch(url["account"], url["user"], url["path"])
# OpenCV PROCESS
image = cv2.imdecode(np.frombuffer("".join(stream), np.uint8), 1)
faces = faceclassifier(image)
# Update the object with new metadatas, number of faces + position
s.object_update(url["account"], url["user"], url["path"], {"face_number" : str(len(faces)), "position" : json.dumps(faces.tolist())})
b.delete(event_id)

20. OpenIO
Open source and proprietary code
OpenIO SDS core
(AGPLv3)
S3 and Swift
Gateway
File Gateway WebUI
Deployment
Tools
Open source Proprietary
Grid For Apps Pro
G4A core
(AGPLv3)

21. OpenIO
Thank You!