1.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
What’s Cooking in the Cloud?
Expanding the Menu for App Developers
Kasper Løvborg Jensen

2.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Chief Wizard, Founder
Leafcastle Labs
Associate Professor
Aarhus University School of Engineering
Organizer
Google Developer Group Aarhus
@kasperljkasper@leafcastlelabs.com

3.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
GCP NEXT ‘16 Conference
• Machine Learning for all!
• Cloud Vision for all!
• Cloud Speech API for all (soon)!
• StackDriver (works with AWS)!
• …lots more!

4.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Outline
• Why cloud?
• Mobile backends
– Experiences
– Patterns
• New powerful Tools/APIs
– Machine Learning
– Speech API
– Cloud Vision
• Demo: TA-800

5.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University

6.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Why I am Excited about Cloud / GCP
• Backends for mobile/wearable/web apps
– Low/No Maintenance
– Scalability
– Reliability
– Security
• APIs / Services for applications
– Integrate/synch with existing Services
– Add powerful features/functionality through APIs
• Great for startups
– Business models and focus on creating value
– Cost scales with growth
– Speed of iteration and experiments

7.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
7

8.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
In My Steps

9.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University

10.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University

11.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
GCP Scalability

12.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
App Backend Needs
• Storage
• Computing
• Network / serving
• Cloud Messaging
• Analytics
• Tools
• …

13.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Google Cloud Platform
Source: GCP NEXT 2016 keynote

14.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Mobile Backends?
Source: Google, Android, Apple, Chrome

15.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Mobile Backends
“Building a backend service for a mobile app is
similar to building a web-based service, with
some additional requirements:
• Limit on-device data storage
• Synchronize data across multiple devices
• Handle the offline case gracefully
• Send notifications and messages
• Minimize battery drain”
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services

16.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Backend Patterns
M.C. Escher

17.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
App Engine w. EndPoints
Modified from: hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services

18.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Pure Firebase
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services

19.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Firebase + App Engine
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services

20.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Firebase + MVMs | Containers
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services

21.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Machine Learning
• Machine Learning
• Vision API
• Speech API
• Translate API
Source: Google

22.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
FROM THIS

23.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
TO THIS

24.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Source: www.tensorflow.org

25.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
But wait, isn’t Machine Learning EVIL?

26.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
If by evil, you mean
AWESOME!

27.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Cloud Speech APi

28.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Speech API
• Solves really hard problems!
– Natural language recognition
– Open vocabulary
– Noisy environments
– 80+ Different languages
hAps://cloud.google.com/speech/

29.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Speech Horror Story ca. 2005
• Distributed speech recognition in
app as part of my PhD
– On PDA running Windows Mobile
with feature extraction through
custom DLL (c/c++)
– Live stream features to Sphinx IV
DSR on Tomcat server that could
only handle one connection/request
at a time due to memory
requirements
– Months of building/training speech,
grammar and acoustic models

30.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University

31.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Live or Batch?
• Batch
– Upload audio sample(s) and get
results when recognition
complete
• Live
– Stream audio to server and
receive partial results as they
become available
hAps://cloud.google.com/speech/

32.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Cloud Vision API

33.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Cloud Vision API
• Solves really hard problems!
– Extract and label objects in arbitrary image
without context
– Can be anything
– Find details like facial features, tilt and
“emotions“
– Lighting conditions
– Blurry pictures
– Vast amount of cameras and settings
– …

34.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Case Study
• Problem: Students falling asleep during
lectures
• Solution: Need person some sort of
contraption to detect students falling
asleep and then wake them up
“Never send a man to
do a machine’s job”
-Gandalf

35.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
TA-800
• IF we wanted to build a Terminator as a
teaching assistant
– Running Android
– Powered by Google Cloud / Skynet
• Detect and “terminate” inactive students
during lectures
– High precision Nerf gatling gun

36.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
TA-800 Hardware
• Nexus 6 or 6p
• ATLAS prototype
• Nerf minigun

37.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
TA-800 Behavior Protocol
While(true){
– Detect situation
• Identify features and classify place/situation
– Acquire potential targets
• Detect if sleeping
– TERMINATE Wake up targets
– Report to master
}

38.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Architecture
Master TA-800

39.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Demo

40.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
TA-800 Hack/Demo
• Image processing based on Cloud Vision API
example app
https://github.com/GoogleCloudPlatform/
cloud-vision/tree/master/android/CloudVision
• Firebase integration based on Firebase chat
example
https://github.com/firebase/AndroidChat
• Total dev time:~6 hours (50%+ time on UI)
• Will be on GitHub:

41.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Cloud Vision API
hAps://cloud.google.com/vision/

42.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Using Cloud Vision API
• Specify request (JSON)
– Image
• Content/source
– Features
• Type
• Max results
• Handle result (JSON)
• Android: use CloudVision API
– com.google.apis:google-api-services-
vision:v1-rev2-1.21.0!

43.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Cloud Vision Detection Types
DETECTION_TYPES = [!
'TYPE_UNSPECIFIED',!
'FACE_DETECTION',!
'LANDMARK_DETECTION',!
'LOGO_DETECTION',!
'LABEL_DETECTION',!
'TEXT_DETECTION',!
'SAFE_SEARCH_DETECTION',!
]!

44.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Rich Response Data as JSON
• Labels
– Things/situations in the image
• Faces
– Faces, location in picture
– Happy, angry, sad, surprised
– Face “landmarks”: eyes, ears, …
– Tilt, …
• Logos
– Known brands and logos
• Landmarks
– “Eiffel Tower”, “AROS museum”, …
– Also including e.g. location metadata
• Texts
– OCR, finding text in the image

45.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
Your turn!
GCP:
https://cloud.google.com/
List of (most) products:
https://cloud.google.com/products/
Playground:
https://cloud-playground.appspot.com/playground/
Launcher:
https://cloud.google.com/launcher/
Firebase:
https://www.firebase.com/

46.
Kasper Løvborg Jensen
Leafcastle Labs | Aarhus University
THANK YOU!
Questions?
@kasperlj
kasper@leafcastlelabs.com