Big data and messaging with Spring

•Download as PPTX, PDF•

5 likes•1,636 views

Artem Bilan

Technology

Artem Bilan
Spring Integration Team

abilan@gopivotal.com

Spring Framework
Spring AMQP
Spring XD
Reactor

https://spring.io/team/artembilan

http://stackoverflow.com/users/2756547

http://www.linkedin.com/in/cleric

https://github.com/artembilan

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Messaging and why do I care?

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Messaging is very simple

Headers

Payload

public static void main(String[] args) {
...
}
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Messaging in Spring IO

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring Integration
Written
Uses

Meet IoC!

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

The heart of Spring Integration
–
–
–

Endpoints (Filters) connected through
Channels (Pipes) exchanging
Message

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring Integration Message Flow

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

How does it work?

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

What is a Big Data application?
 Not just writing a few MR or Hive/Pig jobs.
 The full lifecycle involves
• Ingestion
• Stream Processing
• Workflow Orchestration
• Enterprise Integration
• Export
• Horizontal scalable deployment
 How do you write one of these?
• Status quo is to combine different projects – not ideal
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring XD
• Unified Platform
• Developer Productivity
• Modular Extensibility
• Distributed Architecture
• Portable Runtime
• Hadoop Distribution Agnostic
• Proven Foundation
• XD = ‘eXtreme Data’
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring XD
Files

Sensors

Mobile

Social

Spring XD Shell
Spring XD Runtime

Taps

Jobs

Compute

Workflow

Redis

Ingest

Streams

Export

Export

RDBMS

Gemfire

HDFS

NoSQL

R, SAS

Predictive modeling
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Stream Processing Model

How can
http |we make this|easier?
filter
file

Non-linear stream definitions also supported
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Runtimes
XD Admin

http | filter | file

CLUSTERED NODE

CLUSTERED NODE

CLUSTERED NODE

SINGLE
NODE

HTTP
Module

Filter
Module

File
Module

All
Modules

Rabbit, Redis, (Pluggable)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

In Memory
Transport

http | filter | file

Streams
MessageStore

HTTP
Tail
File
Mail
Twitter
Gemfire
Syslog
TCP
JMS
RabbitMQ
MQTT

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Filter
Transformer
Splitter
Aggregator
Groovy Script
Counters
HTTP
JSON
Java Code

File
HDFS
JDBC
TCP
Mail
RabbitMQ
Gemfire
Splunk
MQTT
Dynamic Router

Taps
 “Listen” to data from another stream
• Other stream is unaffected by the tap and unaware of its presence
• EAI ‘Wiretap’

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Analytics
• Simple Counter
• Field Value Counter
– Count occurrences of named
fields

• Aggregate Counter
– Pre-aggregate counts in time
buckets

• Gauge
– last value

• Rich Gauge
– Last value, running
average, min/max
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

• Abstract API
• Implementations for
– Memory
– Redis

• Looking at
– Pattern
– JPMML

Jobs
 Jobs are a directed graph of steps
 Steps
• Copy or Process data
• Files, Databases, MR, Pig, Hive, Cascading

 Step executions are persisted
• Checkpointing with restart
• Rich error handling capabilities
 Single node or distributed with data partitioning
 Jobs can be Triggered from streams
 Executing jobs generate a stream of event data

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

SPRING XD

Demo

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Learn More. Stay Connected.
•
•
•
•
•
•
•
•

Spring IO: https://spring.io/platform
GitHub: https://github.com/spring-projects
Spring Integration: http://projects.spring.io/spring-integration
Spring XD: http://projects.spring.io/spring-xd
Reactor: https://github.com/reactor
EIP: http://www.eaipatterns.com
Spring Batch: http://projects.spring.io/spring-batch
Spring for Hadoop: http://projects.spring.io/spring-hadoop

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

ありがとう
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Viewers also liked

Presentation a pivotal overviewxKinAnx

Guest Lecture on Big Data in Business, saravana krishnamurthy

Pivotal OSS meetup - MADlib and PivotalRgo-pivotal

Scale Out Your Big Data Apps: The Latest on Pivotal GemFire and GemFire XDVMware Tanzu

Accelerating Operational Excellence in 2015: Calculating the ROI of Real-Time...Catavolt, Inc.

Pivotal Digital Transformation Forum: Data Science Technical OverviewVMware Tanzu

Operationalizing Data AnalyticsVMware Tanzu

Pivotal the new_pivotal_big_data_suite_-_revolutionary_foundation_to_leverage...EMC

Why Are Digital Disruptors Successful And How Can You Become One? VMware Tanzu

Data as the New Oil: Producing Value in the Oil and Gas IndustryVMware Tanzu

Modern Big Data Analytics Tools: An OverviewGreat Wide Open

Women who wrote the analytics book finalmetabrown

Pivotal Data Lake Architecture & its role in security analyticsEMC

Pivotal Digital Transformation Forum: Data Science Bridging the GapVMware Tanzu

Building a Modern Analytic Database with Cloudera 5.8Cloudera, Inc.

Realizing the Promise of Big Data with Hadoop - Cloudera Summer Webinar Serie...Cloudera, Inc.

2014 Big_Data_Forum_PivotalCOMPUTEX TAIPEI

Customer Spotlight: How WellCare Accelerated Big Data Delivery to Improve Ana...VMware Tanzu

Modernizing Architecture for a Complete Data StrategyCloudera, Inc.

Becoming Data-Driven Through Cultural ChangeCloudera, Inc.

Viewers also liked (20)

Presentation a pivotal overview

Guest Lecture on Big Data in Business,

Pivotal OSS meetup - MADlib and PivotalR

Scale Out Your Big Data Apps: The Latest on Pivotal GemFire and GemFire XD

Accelerating Operational Excellence in 2015: Calculating the ROI of Real-Time...

Pivotal Digital Transformation Forum: Data Science Technical Overview

Operationalizing Data Analytics

Pivotal the new_pivotal_big_data_suite_-_revolutionary_foundation_to_leverage...

Why Are Digital Disruptors Successful And How Can You Become One?

Data as the New Oil: Producing Value in the Oil and Gas Industry

Modern Big Data Analytics Tools: An Overview

Women who wrote the analytics book final

Pivotal Data Lake Architecture & its role in security analytics

Pivotal Digital Transformation Forum: Data Science Bridging the Gap

Building a Modern Analytic Database with Cloudera 5.8

Realizing the Promise of Big Data with Hadoop - Cloudera Summer Webinar Serie...

2014 Big_Data_Forum_Pivotal

Customer Spotlight: How WellCare Accelerated Big Data Delivery to Improve Ana...

Modernizing Architecture for a Complete Data Strategy

Becoming Data-Driven Through Cultural Change

Similar to Big data and messaging with Spring

Institutionalizing Open Source - Puneet Sachdev - Nasscom Tech Series - June ...Puneet Sachdev

Building Highly Scalable Spring Applications using In-Memory Data GridsJohn Blum

Asynchronous Event Streams – when java.util.stream met org.osgi.util.promise!...mfrancis

IoT architectureSumit Sharma

Get to know the browser better and write faster web appsLior Bar-On

S2DS London 2015 - Hadoop Real WorldSean Roberts

Data Mesh Part 4 Monolith to MeshJeffrey T. Pollock

Open APIs - Risks and Rewards (Øredev 2013)Nordic APIs

Node summit workshopShubhra Kar

HTTP/2 and a Faster WebC4Media

A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal ...Srivatsan Ramanujam

Operational Machine Learning: Using Microsoft Technologies for Applied Data S...Khalid Salama

Web Architecture - Mechanism and ThreatsSumedt Jitpukdebodin

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Rittman Analytics

What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerceVMware Tanzu

gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20Phil Wilkins

Building a Stock Prediction system with Machine Learning using Geode, SpringX...William Markito Oliveira

Automation + dev ops summit hail hydrate! from stream to lakeTimothy Spann

Modernizing an Existing SOA-based Architecture with APIsApigee | Google Cloud

How to build unified Batch & Streaming Pipelines with Apache Beam and DataflowDaniel Zivkovic

Similar to Big data and messaging with Spring (20)

Institutionalizing Open Source - Puneet Sachdev - Nasscom Tech Series - June ...

Building Highly Scalable Spring Applications using In-Memory Data Grids

Asynchronous Event Streams – when java.util.stream met org.osgi.util.promise!...

IoT architecture

Get to know the browser better and write faster web apps

S2DS London 2015 - Hadoop Real World

Data Mesh Part 4 Monolith to Mesh

Open APIs - Risks and Rewards (Øredev 2013)

Node summit workshop

HTTP/2 and a Faster Web

A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal ...

Operational Machine Learning: Using Microsoft Technologies for Applied Data S...

Web Architecture - Mechanism and Threats

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce

gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20

Building a Stock Prediction system with Machine Learning using Geode, SpringX...

Automation + dev ops summit hail hydrate! from stream to lake

Modernizing an Existing SOA-based Architecture with APIs

How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow

Recently uploaded

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Real Time Object Detection Using Open CVKhem

Architecting Cloud Native ApplicationsWSO2

Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1

Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbuapidays

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...apidays

Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun

MS Copilot expands with MS Graph connectorsNanddeep Nachan

ICT role in 21st century education and its challengesrafiqahmad00786416

FWD Group - Insurer Innovation Award 2024The Digital Insurer

GenAI Risks & Security Meetup 01052024.pdflior mazor

AXA XL - Insurer Innovation Award Americas 2024The Digital Insurer

Manulife - Insurer Transformation Award 2024The Digital Insurer

Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services

2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

Recently uploaded (20)

How to Troubleshoot Apps for the Modern Connected Worker

Real Time Object Detection Using Open CV

Architecting Cloud Native Applications

Boost Fertility New Invention Ups Success Rates.pdf

Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Data Cloud, More than a CDP by Matt Robison

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

Powerful Google developer tools for immediate impact! (2023-24 C)

MS Copilot expands with MS Graph connectors

ICT role in 21st century education and its challenges

FWD Group - Insurer Innovation Award 2024

GenAI Risks & Security Meetup 01052024.pdf

AXA XL - Insurer Innovation Award Americas 2024

Manulife - Insurer Transformation Award 2024

Strategies for Landing an Oracle DBA Job as a Fresher

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Automating Google Workspace (GWS) & more with Apps Script

Big data and messaging with Spring

2. Artem Bilan Spring Integration Team abilan@gopivotal.com Spring Framework Spring AMQP Spring XD Reactor https://spring.io/team/artembilan http://stackoverflow.com/users/2756547 http://www.linkedin.com/in/cleric https://github.com/artembilan © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

10. What is a Big Data application?  Not just writing a few MR or Hive/Pig jobs.  The full lifecycle involves • Ingestion • Stream Processing • Workflow Orchestration • Enterprise Integration • Export • Horizontal scalable deployment  How do you write one of these? • Status quo is to combine different projects – not ideal © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

11. Spring XD • Unified Platform • Developer Productivity • Modular Extensibility • Distributed Architecture • Portable Runtime • Hadoop Distribution Agnostic • Proven Foundation • XD = ‘eXtreme Data’ © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

12. Spring XD Files Sensors Mobile Social Spring XD Shell Spring XD Runtime Taps Jobs Compute Workflow Redis Ingest Streams Export Export RDBMS Gemfire HDFS NoSQL R, SAS Predictive modeling © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

14. Runtimes XD Admin http | filter | file CLUSTERED NODE CLUSTERED NODE CLUSTERED NODE SINGLE NODE HTTP Module Filter Module File Module All Modules Rabbit, Redis, (Pluggable) © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. In Memory Transport http | filter | file

15. Streams MessageStore HTTP Tail File Mail Twitter Gemfire Syslog TCP JMS RabbitMQ MQTT © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. Filter Transformer Splitter Aggregator Groovy Script Counters HTTP JSON Java Code File HDFS JDBC TCP Mail RabbitMQ Gemfire Splunk MQTT Dynamic Router

16. Taps  “Listen” to data from another stream • Other stream is unaffected by the tap and unaware of its presence • EAI ‘Wiretap’ © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

17. Analytics • Simple Counter • Field Value Counter – Count occurrences of named fields • Aggregate Counter – Pre-aggregate counts in time buckets • Gauge – last value • Rich Gauge – Last value, running average, min/max © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. • Abstract API • Implementations for – Memory – Redis • Looking at – Pattern – JPMML

18. Jobs  Jobs are a directed graph of steps  Steps • Copy or Process data • Files, Databases, MR, Pig, Hive, Cascading  Step executions are persisted • Checkpointing with restart • Rich error handling capabilities  Single node or distributed with data partitioning  Jobs can be Triggered from streams  Executing jobs generate a stream of event data © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

20. Learn More. Stay Connected. • • • • • • • • Spring IO: https://spring.io/platform GitHub: https://github.com/spring-projects Spring Integration: http://projects.spring.io/spring-integration Spring XD: http://projects.spring.io/spring-xd Reactor: https://github.com/reactor EIP: http://www.eaipatterns.com Spring Batch: http://projects.spring.io/spring-batch Spring for Hadoop: http://projects.spring.io/spring-hadoop © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Editor's Notes

Unified platform across big data domainsStream processing: ingestion & analyticsBatch processing workflow orchestration & exportProductivityHigh level DSL for managing streams and jobsProven foundation Built on existing assets: Spring Batch, Integration, DataExtensibleDI, Test friendly…DistributedA | B | C - Pluggable transports: Rabbit, Redis, …Portable Runtime Standalone – Simplicity and testabilityYARN –Fault Tolerance and ScalabilityIn-memory data grids – co-location of reference data.PAASHadoop Distribution Agnostic
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
Distributed mode, process boundaries
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
http | filehttp | hdfstwittersearch | hdfsaggregatecounter on hashtags

Big data and messaging with Spring

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Big data and messaging with Spring

Similar to Big data and messaging with Spring (20)

Recently uploaded

Recently uploaded (20)

Big data and messaging with Spring

Editor's Notes