#GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode

•

1 like•2,224 views

In this session we review the design of the current state of support for Apache Geode by Spring Cloud Data Flow, and explore additional use cases and future direction that Spring Cloud Data Flow and Apache Geode might evolve.

Technology

Spring Cloud Data Flow + Geode
Sabby Anandan | Product Manager | @sabbyanandan

Stream Batch
Spring Cloud Data Flow
Spring Cloud Stream Spring Cloud Task
Shell; DSL; REST-
APIs
Drag & Drop UI Security
OOTB
Connectors
Reactive Data Science

Dataﬂow
Server
Admin / Flo UI
Shell
CURL
??X
Stream/Task Spring Boot Apps
YARN

Why are we here?
Data Pipelines requiring:
• Low latency and in-memory processing
• High Throughput SLAs
• Correlation between reference-data and data-
in-ﬂight
• Frequent data-shufﬂing

Region Data Buckets
http | transform | log
Geode Cluster
transform-processor.jar
PARTITION
transform-processor.jar
PARTITION
log-sink.jar
PARTITION
log-sink.jar
PARTITION
http-source.jar
PARTITION_PROXY
http-source.jar
PARTITION_PROXY

What’s next?
•+/- scaling and automatic re-partitioning
•Stream / Task metadata-repository
•Key-Value store for OOTB Counters
•Partition level local-state and SQL-like
stream processing

11
Join the Apache Geode Community!
• Check out http://geode.incubator.apache.org
• Subscribe: user-subscribe@geode.incubator.apache.org
• Download: http://geode.incubator.apache.org/releases/

What's hot

#GeodeSummit - Redis to Geode AdaptorPivotalOpenSourceHub

Spark on MesosJen Aman

Operationalizing YARN based Hadoop Clusters in the CloudDataWorks Summit/Hadoop Summit

October 2016 HUG: Architecture of an Open Source RDBMS powered by HBase and ...Yahoo Developer Network

Presto on Apache Spark: A Tale of Two Computation EnginesDatabricks

Operational Tips for Deploying Spark by Miklos ChristineSpark Summit

Building Efficient Pipelines in Apache SparkJeremy Beard

Building Effective Near-Real-Time Analytics with Spark Streaming and KuduJeremy Beard

Simplified Cluster Operation & TroubleshootingDataWorks Summit/Hadoop Summit

HDInsight for ArchitectsAshish Thapliyal

Achieve big data analytic platform with lambda architecture on cloudScott Miao

5 Apache Spark Tips in 5 MinutesCloudera, Inc.

IMCSummit 2015 - Day 2 Developer Track - Anatomy of an In-Memory Data Fabric:...In-Memory Computing Summit

Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared ClustersDataWorks Summit

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程HBaseCon

Apache sqoop with an use caseDavin Abraham

Getting Started with Apache Cassandra and Apache Zeppelin (DuyHai DOAN, DataS...DataStax

Apache Geode Offheap StoragePivotalOpenSourceHub

Apache Kylin: Speed Up Cubing with Apache Spark with Luke Han and Shaofeng ShiDatabricks

Spark Uber Development KitDataWorks Summit/Hadoop Summit

What's hot (20)

#GeodeSummit - Redis to Geode Adaptor

Spark on Mesos

Operationalizing YARN based Hadoop Clusters in the Cloud

October 2016 HUG: Architecture of an Open Source RDBMS powered by HBase and ...

Presto on Apache Spark: A Tale of Two Computation Engines

Operational Tips for Deploying Spark by Miklos Christine

Building Efficient Pipelines in Apache Spark

Building Effective Near-Real-Time Analytics with Spark Streaming and Kudu

Simplified Cluster Operation & Troubleshooting

HDInsight for Architects

Achieve big data analytic platform with lambda architecture on cloud

5 Apache Spark Tips in 5 Minutes

IMCSummit 2015 - Day 2 Developer Track - Anatomy of an In-Memory Data Fabric:...

Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程

Apache sqoop with an use case

Getting Started with Apache Cassandra and Apache Zeppelin (DuyHai DOAN, DataS...

Apache Geode Offheap Storage

Apache Kylin: Speed Up Cubing with Apache Spark with Luke Han and Shaofeng Shi

Spark Uber Development Kit

Viewers also liked

#GeodeSummit: Architecting Data-Driven, Smarter Cloud Native Apps with Real-T...PivotalOpenSourceHub

#GeodeSummit - Wall St. Derivative Risk Solutions Using GeodePivotalOpenSourceHub

#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)PivotalOpenSourceHub

#GeodeSummit: Easy Ways to Become a Contributor to Apache GeodePivotalOpenSourceHub

#GeodeSummit: Combining Stream Processing and In-Memory Data Grids for Near-R...PivotalOpenSourceHub

#GeodeSummit - Using Geode as Operational Data Services for Real Time Mobile ...PivotalOpenSourceHub

#GeodeSummit - Modern manufacturing powered by Spring XD and GeodePivotalOpenSourceHub

#GeodeSummit - Large Scale Fraud Detection using GemFire Integrated with Gree...PivotalOpenSourceHub

Redis, a 2 minutes introductionMirko Calvaresi

Pivotal Cloud Foundry: A Technical OverviewVMware Tanzu

Building Services with WSO2 Application Server and WSO2 Microservices Framewo...Sagara Gunathunga

Cloud Native Runtime PlatformVMware Tanzu

Spring Cloud Into ProductionTodd Miller

Devops Recto-Verso @ DevoxxMAArnaud Héritier

WSO2ConUS 2015 - Introduction to WSO2 Microservices Server (MSS)Afkham Azeez

QCon SP 2016 - Construindo Microservices Auto-curáveis com Spring Cloud e Net...Rodrigo Cândido da Silva

Spring Cloud Servicesの紹介 #pcf_tokyoToshiaki Maki

Apache Geode Meetup, LondonApache Geode

Build your first Internet of Things app today with Open SourceApache Geode

Viewers also liked (19)

#GeodeSummit: Architecting Data-Driven, Smarter Cloud Native Apps with Real-T...

#GeodeSummit - Wall St. Derivative Risk Solutions Using Geode

#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)

#GeodeSummit: Easy Ways to Become a Contributor to Apache Geode

#GeodeSummit: Combining Stream Processing and In-Memory Data Grids for Near-R...

#GeodeSummit - Using Geode as Operational Data Services for Real Time Mobile ...

#GeodeSummit - Modern manufacturing powered by Spring XD and Geode

#GeodeSummit - Large Scale Fraud Detection using GemFire Integrated with Gree...

Redis, a 2 minutes introduction

Pivotal Cloud Foundry: A Technical Overview

Building Services with WSO2 Application Server and WSO2 Microservices Framewo...

Cloud Native Runtime Platform

Spring Cloud Into Production

Devops Recto-Verso @ DevoxxMA

WSO2ConUS 2015 - Introduction to WSO2 Microservices Server (MSS)

QCon SP 2016 - Construindo Microservices Auto-curáveis com Spring Cloud e Net...

Spring Cloud Servicesの紹介 #pcf_tokyo

Apache Geode Meetup, London

Build your first Internet of Things app today with Open Source

Similar to #GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode

Cowboy Dating with Big Data or DWH Evolution in Action, Борис ТрофимовSigma Software

Unified Batch & Stream Processing with Apache SamzaDataWorks Summit

Stateful Interaction In Serverless Architecture With Redis: Pyounguk ChoRedis Labs

Flink Forward Berlin 2018: Xiaowei Jiang - Keynote: "Unified Engine for Data ...Flink Forward

Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...Lace Lofranco

Case Study: Stream Processing on AWS using Kappa ArchitectureJoey Bolduc-Gilbert

Apache Flink: Past, Present and FutureGyula Fóra

SharePoint Performance Optimization In 10 Steps for the IT ProfessionalJoel Oleson

What's New in Apache Hive 3.0?DataWorks Summit

What's New in Apache Hive 3.0 - TokyoDataWorks Summit

Speed Up Your Queries with Hive LLAP Engine on Hadoop or in the Cloudgluent.

Otimizações de Projetos de Big Data, Dw e AI no Microsoft AzureLuan Moreno Medeiros Maciel

A Real World Guide to Building Highly Available Fault Tolerant SharePoint FarmsEric Shupps

How to Build Real-Time Streaming Analytics with an In-memory, Scale-out SQL D...VoltDB

Distributed Data Quality - Technical Solutions for Organizational ScalingJustin Cunningham

Building Fast Applications for Streaming Datafreshdatabos

Using the Power of Big SQL 3.0 to Build a Big Data-Ready Hybrid WarehouseRizaldy Ignacio

Cowboy dating with big data, Борис ТрофімовSigma Software

Building Super Fast Cloud-Native Data Platforms - Yaron Haviv, KubeCon 2017 EUYaron Haviv

SharePoint 2010 Boost your farm performance!Brian Culver

Similar to #GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode (20)

Cowboy Dating with Big Data or DWH Evolution in Action, Борис Трофимов

Unified Batch & Stream Processing with Apache Samza

Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho

Flink Forward Berlin 2018: Xiaowei Jiang - Keynote: "Unified Engine for Data ...

Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...

Case Study: Stream Processing on AWS using Kappa Architecture

Apache Flink: Past, Present and Future

SharePoint Performance Optimization In 10 Steps for the IT Professional

What's New in Apache Hive 3.0?

What's New in Apache Hive 3.0 - Tokyo

Speed Up Your Queries with Hive LLAP Engine on Hadoop or in the Cloud

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

A Real World Guide to Building Highly Available Fault Tolerant SharePoint Farms

How to Build Real-Time Streaming Analytics with an In-memory, Scale-out SQL D...

Distributed Data Quality - Technical Solutions for Organizational Scaling

Building Fast Applications for Streaming Data

Using the Power of Big SQL 3.0 to Build a Big Data-Ready Hybrid Warehouse

Cowboy dating with big data, Борис Трофімов

Building Super Fast Cloud-Native Data Platforms - Yaron Haviv, KubeCon 2017 EU

SharePoint 2010 Boost your farm performance!

Recently uploaded

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Principled Technologies

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous

A Domino Admins Adventures (Engage 2024)Gabriella Davis

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer

Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi

🐬 The future of MySQL is Postgres 🐘RTylerCroy

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93

Artificial Intelligence: Facts and MythsJoaquim Jorge

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1

Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services

Real Time Object Detection Using Open CVKhem

Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Manulife - Insurer Innovation Award 2024The Digital Insurer

Recently uploaded (20)

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Boost PC performance: How more available memory can improve productivity

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

A Domino Admins Adventures (Engage 2024)

How to Troubleshoot Apps for the Modern Connected Worker

presentation ICT roal in 21st century education

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

🐬 The future of MySQL is Postgres 🐘

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Artificial Intelligence: Facts and Myths

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Boost Fertility New Invention Ups Success Rates.pdf

Strategies for Landing an Oracle DBA Job as a Fresher

Real Time Object Detection Using Open CV

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Manulife - Insurer Innovation Award 2024

#GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode

1. Spring Cloud Data Flow + Geode Sabby Anandan | Product Manager | @sabbyanandan

2. Stream Batch Spring Cloud Data Flow Spring Cloud Stream Spring Cloud Task Shell; DSL; REST- APIs Drag & Drop UI Security OOTB Connectors Reactive Data Science

3. Dataﬂow Server Admin / Flo UI Shell CURL ??X Stream/Task Spring Boot Apps YARN

4. Why are we here? Data Pipelines requiring: • Low latency and in-memory processing • High Throughput SLAs • Correlation between reference-data and data- in-ﬂight • Frequent data-shufﬂing

5. http | transform | log

6. | = ? http | transform | log

7. | = Binder

8. Binders

9. Region Data Buckets http | transform | log Geode Cluster transform-processor.jar PARTITION transform-processor.jar PARTITION log-sink.jar PARTITION log-sink.jar PARTITION http-source.jar PARTITION_PROXY http-source.jar PARTITION_PROXY

10. What’s next? •+/- scaling and automatic re-partitioning •Stream / Task metadata-repository •Key-Value store for OOTB Counters •Partition level local-state and SQL-like stream processing

11. 11 Join the Apache Geode Community! • Check out http://geode.incubator.apache.org • Subscribe: user-subscribe@geode.incubator.apache.org • Download: http://geode.incubator.apache.org/releases/

#GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to #GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode

Similar to #GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode (20)

More from PivotalOpenSourceHub

More from PivotalOpenSourceHub (14)

Recently uploaded

Recently uploaded (20)

#GeodeSummit - Integration & Future Direction for Spring Cloud Data Flow & Geode