Integrate Machine Learning into Your Spring Application in Less than an Hour

1© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | 1© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved |
With Deep Java Library
Hermann Burgmeier
Mikhail Shapirov
Qing Lan
Vaibhav Goel
Integrate Machine Learning into
Your Spring Boot Application in
Less than an Hour

2© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Put machine learning in the hands of all developers

MACHINE LEARNING IN 2020
• ML is a top priority for many organizations in 2020
• ML pulls through more compute and data
(it can pull it away to another cloud too)
• Tens of thousands of customers are already running ML on AWS. It’s real
and happening now in your accounts.

COMMON CHALLENGES
• Skills gap—not enough people can build ML models
• ML model building is a time-consuming and complex process
• Finding the right business use cases that could benefit from ML

Easier to build Easier to scale Easier to apply

The AWS ML Stack
Broadest and most complete set of Machine Learning capabilities
VISION SPEECH TEXT SEARCH CHATBOTS PERSONALIZATION FORECASTING FRAUD DEVELOPMENT CONTACT CENTERS
Ground
Truth
Augmented
AI
ML
Marketplace
Neo
Built-in
algorithms
Notebooks Experiments
Model
training &
tuning
Debugger Autopilot
Model
hosting
Model Monitor
Deep Learning
AMIs & Containers
GPUs &
CPUs
Elastic
Inference
Inferentia
(Inf1 instance)
FPGA
Amazon
Rekognition
Amazon
Polly
Amazon
Transcribe
+Medical
Amazon
Comprehend
+Medical
Amazon
Translate
Amazon
Lex
Amazon
Personalize
Amazon
Forecast
Amazon
Fraud Detector
Amazon
CodeGuru
AI SERVICES
ML SERVICES
ML FRAMEWORKS & INFRASTRUCTURE
Amazon
Textract
Amazon
Kendra
Contact Lens
For Amazon Connect
SageMaker Studio IDE
NEW
NEW! NEW! NEW! NEW!
NEW!
NEW! NEW! NEW! NEW! NEW!
Amazon SageMaker

Deep Learning in Java with DJL

DJL – The Machine Learning framework in Java
• API - Designed for Java
• Multithread support and memory control
• Engine Agnostic: Write Once, Run Anywhere (MXNet, PyTorch, TensorFlow)
• ModelZoo: 70+ pretrained models for out of box deployment
• AWS actively contributes to Open Source ML projects

DJL – Build AI Application with no ML Experience
• DJL on Android device
• Easy Deployment
• 70+ models out of box
deployment in applications

Why should you use DJL?
• Most enterprise customers using Java
• Existing Java API for Deep Learning is painful
• Deploy ML in Java is challenging - Hard to maintain
• Java community lacks DL standard
• No single standard in Java DL packages
• DL with strongly typed language
• Service code - readability and reliability

DJL – Key advantage
• Easy: 10 line inference setup
• Minimum dependency requirement
• Fast: up to 2x performance boost on small model inference. Used by Ads,
query understanding and image classification
• Large scale: Offline data processing with 800M inference load on Apache
Spark. Online streaming real-time processing (AWS Kinesis, Apache Flink …)
• Stable: 100+ hour benchmark test on continuous inference call.

DJL – Easy Interface for Java Developers
• DJL Live Jshell
• DJL interactive console

DJL for training
• Engine ready: MXNet and PyTorch
• Built-in metrics: Built-in metrics system to easy collect training information
• Dataset ready: Built-in dataset for CV and NLP
• Model ready: Built-in CV and NLP models
• Multi-GPU support: Support training on Multi-GPU
• Transfer learning: Use your existing models to do transfer learning
• Jupyter Notebook Support (CPU/GPU)

DJL Customer Study

Amazon Advertising
• Multiple offerings on and off Amazon pages
• Sponsored ads (Sponsored Brands, Sponsored Display, Sponsored Products,
Stores)
• Display ads
• Video ads
• Amazon DSP
• Sponsored Products
• Help shoppers discover and find products across Amazon's vast
product selection.
• Advertisers promote their products on search result and product
pages to help increase reach and grow their business.

Sponsored Products

Machine Learning in Sponsored Products
• 100s of machine learning (ML) models deployed globally for many use
cases like click-through rate prediction.
• More use cases require ML in advertising. New data science teams and
models emerge constantly.
• Different teams prefer different ML frameworks.
• Models are hosted in a Java-based model hosting service.
• Modern software engineering techniques like strong typing and
dependency Injection.
• Java stack is well integrated in the operational safety frameworks.

DJL Usage in Sponsored Products
• With DJL the scientists building the model can choose the best ML
framework for them and their needs.
• New ML frameworks supported by DJL can be easily integrated into our
stack.
• DJL enables us to run models built with different ML frameworks side-
by-side in the same JVM without infrastructure changes.
• DJL Translators allows us to keep feature generation independent of ML
framework used for a model.

Amazon Customer Behavior Analytics
• Provides best-in-class targeting for any Amazon product or service to drive
customer engagement
• Helps to provide personalized marketing to customers
• Email campaigns
• Social Ads
• Website banners
• Identify customer’s propensity to take action in various product categories

Machine Learning in Customer Behavior Analytics
• Uses a multi-label classification model in MXNet for building propensity models
• A single model is used for thousand of product categories
• Use high dimensional feature data
• Hundreds of Thousands of features per customer
• The team consists of a mix of software development engineers and research scientists
• Science team prefers using Python
• Engineering team prefers using Java/Scala

DJL Usage in Customer Behavior Analytics
• With DJL
• Science team can use MXNet Python APIs to build the model
• Engineering team uses DJL to run inference using Apache Spark with
Scala
• DJL decoupled our system from ML framework
• DJL easily handled high dimensional data
• DJL reduced batch inference time by 85%
• From 24 hours to 3.5 hours

DJL Spring Boot Starter
• Provides more native experience for Spring Developers
• Handles dependency management for inference
• Auto-configuration option for model look up and predictor wiring

DJL Spring Boot Dependency Management
<parent>
<artifactId>spring-boot-starter-parent</artifactId>
<groupId>org.springframework.boot</groupId>
</parent>
<properties>
<jna.version>5.3.0</jna.version> 
</properties>
<dependencies>
<dependency>
<groupId>ai.djl.spring</groupId>
<artifactId>djl-spring-boot-starter-mxnet-linux_86_64</artifactId>
<version>{djl.version}</version>
</dependency>
</dependencies>

DJL Spring Boot Auto <Platform> Dependency
plugins {
id("org.springframework.boot")
}
repositories {
mavenCentral()
}
dependencies {
implementation("ai.djl.spring:djl-spring-boot-starter-pytorch-auto")
...
}

DJL Spring Boot Auto-Configuration
Maven:
<dependency>
<groupId>ai.djl.spring</groupId>
<artifactId>djl-spring-boot-starter-autoconfigure</artifactId>
<version>{djl.version}</version>
</dependency>
Gradle:
dependencies {
implementation("ai.djl.spring:djl-spring-boot-starter-autoconfigure
:${djl.starter.version}")
...
}

DJL Spring Boot Configuration
- application.yml/application.properties
Access to the model zoo allows selection of application types (or inference
types) from the following set:
• QUESTION_ANSWER(NLP.QUESTION_ANSWER),
TEXT_CLASSIFICATION(NLP.TEXT_CLASSIFICATION),
IMAGE_CLASSIFICATION(CV.IMAGE_CLASSIFICATION),
OBJECT_DETECTION(CV.OBJECT_DETECTION),
ACTION_RECOGNITION(CV.ACTION_RECOGNITION),
INSTANCE_SEGMENTATION(CV.INSTANCE_SEGMENTATION),
POSE_ESTIMATION(CV.POSE_ESTIMATION),
SEMANTIC_SEGMENTATION(CV.SEMANTIC_SEGMENTATION);

DJL Spring Boot Configuration
djl:
# Define application type
application-type: OBJECT_DETECTION
# Define input data type, a model may accept multiple input data
type
input-class: ai.djl.modality.cv.Image
# Define output data type, a model may generate different out put
output-class: ai.djl.modality.cv.output.DetectedObjects
# Define filters that matches your application's need
model-filter:
size: 512
backbone: mobilenet1.0
# Override default pre-processing/post-processing behavior
arguments:
threshold: 0.7

DJL Spring Boot: Using Predictors
Inject predictor given for the specified model and input/output:
@Resource
private Supplier<Predictor<Image, DetectedObjects>> predictorProvider;
@Resource
private ImageFactory imageFactory; // convenience factory for images
Use with try-with-resources:
try (var predictor = predictorProvider.get()) {
var results = predictor.predict(imageFactory.fromFile(("/myimage.jpg")));
for(var result : results.items()) {
LOG.info("results: {}", result.toString());
}
}

Containerize your Spring Boot Application
jib {
from.image = "adoptopenjdk/openjdk13:debian"
to.image = "{repository-spec}/djl- spring-boot-app"
to.tags = setOf(version.toString().plus("-")
.plus(inferredClassifier).plus("-")
.plus(commitHash))
}
---
$./gradlew jib

CD Cycle
1. Build
2. Upload to Container Registry
3. Run on Kubernetes (staged)

GitOps with EKS and Team Collaboration
ArgoCD
GitHub Repos
App of apps

DJL Spring Boot: What is next?
- Integrate DJL functionality
- Include training (and transfer learning) with DJL Spring Boot Starter
- Create solutions and workshops with full CI/CD and training/inference
feedback loop and multi-tenancy
- Running on EKS with GPU optimized instances

DJL - Future
• Android Support: Creating Android-based training and inference apps with
DJL
• Book – Dive into Deep Learning (Java Edition)
• Integrate with more AWS Services (EMR, Lambda, etc…)
• Federate Learning with DJL (Work in Progress)
• Reinforcement Learning with DJL (work in Progress)
• Distributed Inference on GPU support

References
Resources and Documentation:
https://djl.ai/
https://docs.djl.ai/
Source Code:
https://github.com/awslabs/djl/
https://github.com/awslabs/djl-spring-boot-starter
https://github.com/awslabs/djl-spring-boot-starter-demo

Integrate Machine Learning into Your Spring Application in Less than an Hour

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Integrate Machine Learning into Your Spring Application in Less than an Hour

Ähnlich wie Integrate Machine Learning into Your Spring Application in Less than an Hour (20)

Mehr von VMware Tanzu

Mehr von VMware Tanzu (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Integrate Machine Learning into Your Spring Application in Less than an Hour