SlideShare ist ein Scribd-Unternehmen logo
1 von 7
Downloaden Sie, um offline zu lesen
SAP Lambda Architecture Point Of View
SAP Digital Enterprise Platform
Doug Martino
January 2016 External
Inside cover
SAP External | 3
Introduction
New Architectures for Big Data
Whether MVC for user interfaces, or Spine and Leaf for data centers, new
architecture patterns in our industry act as sort of historical markers of the
effectiveness and acceptance of new technologies. Practical techniques push the
bounds resulting in a shift. Application of distributed storage and streaming
capabilities such as Kafka and of course Hadoop are shifting Big Data architectures
from a layer cake concept, or North/South oriented approach to one which can be
thought of as an East/West architectural concept. Recently popular is Lambda
Architecture, this article presents an SAP HANA based rendering of the Lambda
Architecture.
Lambda Architecture
Before discussing how SAP products can be used in a Lambda fashion, a brief
overview of Lambda is in order. Attribution must go to Nathan Marz for his work.
Typically consisting of three components, a batch layer, a speed layer, and a
serving layer, Lambda rethinks how data flows through an analytics system. Data
flows left to right in a multi speed architecture; with all data stored in the batch layer
and the speed layer simultaneously, while the serving layer is used to combine and
analyze that data as required. Advantages are fault tolerance, scalability, while at
the same time offering multiple latencies to supporting a a variety of performance
requirements.
With Lambda some business questions are answered using the speed layer, others
answered by the batch layer, and still others answered by combing both. As
algorithms in the speed layer decide which data to keep, answer immediate
questions such as which ad to display or how much discount to offer, it also stores
an immutable copy of the data into the batch layer. In addition to acting as a replay
source, data in the batch layer is suited for use cases such as long tail analytics,
machine learning, or propensity to ‘act’ types of endeavors. Fault tolerance,
resilience, ultimate scale, and back testing are derived from having the immutable
copy of the original data for replay through the downstream components. The
serving layer may persist a copy of the data to be used for additional downstream
use of the data such as dashboards or feeds to another system. Shown here are
several open source options for the components which make up a Lambda
Architecture.
SAP External | 4
Traditional Lambda Architecture
Lambda with SAP HANA Platform
At SAP we are rapidly delivering value with our customers by employing these types of architecture patterns.
In 2012 we started moving the HANA product towards the SAP Real-time data architecture where one already
finds the notions of Lambda. Smart Data Streaming is the obvious choice for the real-time layer. Born on Wall
Street, this component is capable of ingesting millions of rows per second into HANA. Complete with a rich set
of API’s, stream and window compute constructs, and scale out capability, it can send the right data into
HANA and all the data into HDFS as necessary.
As the leading in-memory database platform, HANA meets the needs of the serving component of Lambda
rather well. The data can be stored once, then transformed, aggregated, and calculated dynamically. Easy to
understand in-memory compute engines such as graph, spatial, OLAP, and predictive analytics are available
to manipulate the data where it is stored. Because the immutable data is kept in an append only columnar
SQL database and the serving is done on the fly, our customer’s experience a valuable and unique
combination of expressive, declarative programming capability on data which is managed in a SQL
framework. This allows data access via everyday BI tools via standard ODBC or JDBC SQL as well as ODBO
MDX. Based on Node.js, the HANA app server XS Advanced provides amongst other things noSQL access to
the data via restful ODATA.
SAP External | 5
Often times requirements change such that new algorithms or new derivations of the immutable data are
required. Because of the dynamic nature of materialization employed by the HANA core architecture, these
changes can be made almost instantly. This flexibility changes the logistics of replay, allowing for rapid
experimentation and moves into a production environment. In the case that a schema needs to be extended
HANA does have schema flexibility whereby columns may be added to an existing table structure, leaving any
existing code intact. The amount of flexibility offered by HANA once the immutable data has been stored is a
fundamental differentiator of HANA. This flexibility offers reduced time to value and increased data agility.
While the HANA platform is working extremely well for real-time applications, Lambda is complete by
incorporating Vora. From an architecture standpoint, there are several aspects to add to the discussion. First
is the aspect of scalability.
SAP External | 6
One of the beauties of the lambda is scaling each component dynamically and independently as latency
requirements shift to meet changing business requirements. This notion of independent scaling allows
appropriate resources allocation in a fit-for purpose model. Vora’s use of Hadoop moves the scale of the
overall system up to multi petabyte range. More importantly, addressing long tail analytics and other big data
compute problems are now possible.
This leads to the next concept, which is one of throughput and algorithmic capability. Each of the real-
time/batch/serving layers are best suited for a particular algorithm at a given throughout. All of them can run
‘out of band’ code; the question is at which level of concurrency and complexity. Rather than have JAVA JVM
as the foundation of each, SAP have the appropriate engines with well-documented and supported extension
capabilities. One may argue that open source provides the ultimate extension capability but only as a
consuming organization is willing to merge code. In this case, SAP is providing high performance, fit for
purpose engines to achieve the desired results more efficiently.
This leaves us with the replay topic, an oft forgotten concept, especially in an operational batch context.
Usually left for that – operations, it is often the case that re-running loads into a system to catch up with the
real world, or to modify results based on new algorithms cause over provisioning of hardware in today's ETL
and serving layers. By accounting for this capability up front, the opportunity to make transforms happen at the
right time reduce the time and energy required to keep things in phase. HANA is a fantastic example of this
given its powerful late materialization capability. Eventually elastic capabilities will further reduce replay
workload stress by optimal allocation of compute resources with finer grain and lower latency. By employing
lambda and other emerging big data architecture patterns, our customers remain well positioned as the state
of the art continues forward.
www.sap.com/contactsap
© 2016 SAP SE or an SAP affiliate company.
All rights reserved.
No part of this publication may be reproduced or transmitted in any
form or for any purpose without the express permission of SAP SE
or an SAP affiliate company.
SAP and other SAP products and services mentioned herein as well
as their respective logos are trademarks or registered trademarks of
SAP SE (or an SAP affiliate company) in Germany and other
countries. Please see http://www.sap.com/corporate-
en/legal/copyright/index.epx#trademark for additional trademark
information and notices. Some software products marketed by SAP
SE and its distributors contain proprietary software components of
other software vendors.
National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate
company for informational purposes only, without representation or
warranty of any kind, and SAP SE or its affiliated companies shall
not be liable for errors or omissions with respect to the materials.
The only warranties for SAP SE or SAP affiliate company products
and services are those that are set forth in the express warranty
statements accompanying such products and services, if any.
Nothing herein should be construed as constituting an additional
warranty.
In particular, SAP SE or its affiliated companies have no obligation to
pursue any course of business outlined in this document or any
related presentation, or to develop or release any functionality
mentioned therein. This document, or any related presentation, and
SAP SE’s or its affiliated companies’ strategy and possible future
developments, products, and/or platform directions and functionality
are all subject to change and may be changed by SAP SE or its
affiliated companies at any time for any reason without notice. The
information in this document is not a commitment, promise, or legal
obligation to deliver any material, code, or functionality. All forward-
looking statements are subject to various risks and uncertainties that
could cause actual results to differ materially from expectations.
Readers are cautioned not to place undue reliance on these forward-
looking statements, which speak only as of their dates, and they
should not be relied upon in making purchasing decisions.

Weitere ähnliche Inhalte

Was ist angesagt?

SplunkSummit 2015 - Real World Big Data Architecture
SplunkSummit 2015 -  Real World Big Data ArchitectureSplunkSummit 2015 -  Real World Big Data Architecture
SplunkSummit 2015 - Real World Big Data ArchitectureSplunk
 
Intelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff PollockIntelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff PollockJeffrey T. Pollock
 
Operational Analytics Using Spark and NoSQL Data Stores
Operational Analytics Using Spark and NoSQL Data StoresOperational Analytics Using Spark and NoSQL Data Stores
Operational Analytics Using Spark and NoSQL Data StoresDATAVERSITY
 
In-Memory Database Platform for Big Data
In-Memory Database Platform for Big DataIn-Memory Database Platform for Big Data
In-Memory Database Platform for Big DataSAP Technology
 
Flexpod with SAP HANA and SAP Applications
Flexpod with SAP HANA and SAP ApplicationsFlexpod with SAP HANA and SAP Applications
Flexpod with SAP HANA and SAP ApplicationsLishantian
 
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...Igor De Souza
 
Big Data & Data Lakes Building Blocks
Big Data & Data Lakes Building BlocksBig Data & Data Lakes Building Blocks
Big Data & Data Lakes Building BlocksAmazon Web Services
 
Hadoop Powers Modern Enterprise Data Architectures
Hadoop Powers Modern Enterprise Data ArchitecturesHadoop Powers Modern Enterprise Data Architectures
Hadoop Powers Modern Enterprise Data ArchitecturesDataWorks Summit
 
IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...
IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...
IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...Mark Rittman
 
Next Generation Enterprise Architecture
Next Generation Enterprise ArchitectureNext Generation Enterprise Architecture
Next Generation Enterprise ArchitectureMapR Technologies
 
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017Lviv Startup Club
 
Designing the Next Generation Data Lake
Designing the Next Generation Data LakeDesigning the Next Generation Data Lake
Designing the Next Generation Data LakeRobert Chong
 
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Rittman Analytics
 
Big Data, Big Thinking: Simplified Architecture Webinar Fact Sheet
Big Data, Big Thinking: Simplified Architecture Webinar Fact SheetBig Data, Big Thinking: Simplified Architecture Webinar Fact Sheet
Big Data, Big Thinking: Simplified Architecture Webinar Fact SheetSAP Technology
 
Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)Jeffrey T. Pollock
 
Big Data and BI Tools - BI Reporting for Bay Area Startups User Group
Big Data and BI Tools - BI Reporting for Bay Area Startups User GroupBig Data and BI Tools - BI Reporting for Bay Area Startups User Group
Big Data and BI Tools - BI Reporting for Bay Area Startups User GroupScott Mitchell
 
The Impact of SAP Hana on the SAP Infrastructure Utility Services Marketplace
The Impact of SAP Hana on the SAP Infrastructure Utility Services MarketplaceThe Impact of SAP Hana on the SAP Infrastructure Utility Services Marketplace
The Impact of SAP Hana on the SAP Infrastructure Utility Services MarketplaceLisa Milani, MBA
 
Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...
Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...
Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...DATAVERSITY
 
Solving Performance Problems on Hadoop
Solving Performance Problems on HadoopSolving Performance Problems on Hadoop
Solving Performance Problems on HadoopTyler Mitchell
 

Was ist angesagt? (19)

SplunkSummit 2015 - Real World Big Data Architecture
SplunkSummit 2015 -  Real World Big Data ArchitectureSplunkSummit 2015 -  Real World Big Data Architecture
SplunkSummit 2015 - Real World Big Data Architecture
 
Intelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff PollockIntelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff Pollock
 
Operational Analytics Using Spark and NoSQL Data Stores
Operational Analytics Using Spark and NoSQL Data StoresOperational Analytics Using Spark and NoSQL Data Stores
Operational Analytics Using Spark and NoSQL Data Stores
 
In-Memory Database Platform for Big Data
In-Memory Database Platform for Big DataIn-Memory Database Platform for Big Data
In-Memory Database Platform for Big Data
 
Flexpod with SAP HANA and SAP Applications
Flexpod with SAP HANA and SAP ApplicationsFlexpod with SAP HANA and SAP Applications
Flexpod with SAP HANA and SAP Applications
 
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
 
Big Data & Data Lakes Building Blocks
Big Data & Data Lakes Building BlocksBig Data & Data Lakes Building Blocks
Big Data & Data Lakes Building Blocks
 
Hadoop Powers Modern Enterprise Data Architectures
Hadoop Powers Modern Enterprise Data ArchitecturesHadoop Powers Modern Enterprise Data Architectures
Hadoop Powers Modern Enterprise Data Architectures
 
IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...
IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...
IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...
 
Next Generation Enterprise Architecture
Next Generation Enterprise ArchitectureNext Generation Enterprise Architecture
Next Generation Enterprise Architecture
 
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
 
Designing the Next Generation Data Lake
Designing the Next Generation Data LakeDesigning the Next Generation Data Lake
Designing the Next Generation Data Lake
 
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
 
Big Data, Big Thinking: Simplified Architecture Webinar Fact Sheet
Big Data, Big Thinking: Simplified Architecture Webinar Fact SheetBig Data, Big Thinking: Simplified Architecture Webinar Fact Sheet
Big Data, Big Thinking: Simplified Architecture Webinar Fact Sheet
 
Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)
 
Big Data and BI Tools - BI Reporting for Bay Area Startups User Group
Big Data and BI Tools - BI Reporting for Bay Area Startups User GroupBig Data and BI Tools - BI Reporting for Bay Area Startups User Group
Big Data and BI Tools - BI Reporting for Bay Area Startups User Group
 
The Impact of SAP Hana on the SAP Infrastructure Utility Services Marketplace
The Impact of SAP Hana on the SAP Infrastructure Utility Services MarketplaceThe Impact of SAP Hana on the SAP Infrastructure Utility Services Marketplace
The Impact of SAP Hana on the SAP Infrastructure Utility Services Marketplace
 
Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...
Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...
Webinar: Data Modeling and Shortcuts to Success in Scaling Time Series Applic...
 
Solving Performance Problems on Hadoop
Solving Performance Problems on HadoopSolving Performance Problems on Hadoop
Solving Performance Problems on Hadoop
 

Ähnlich wie SAP Lambda Architecture Point of View

209 hana-defining-capability-whitepaper
209 hana-defining-capability-whitepaper209 hana-defining-capability-whitepaper
209 hana-defining-capability-whitepaperbbenthach
 
Analytics Products L2 public 2020-23 Black.pptx
Analytics Products L2 public 2020-23 Black.pptxAnalytics Products L2 public 2020-23 Black.pptx
Analytics Products L2 public 2020-23 Black.pptxBurakAyan6
 
Strategies To Overcome SAP S/4HANA Data Migration Challenges
Strategies To Overcome SAP S/4HANA Data Migration ChallengesStrategies To Overcome SAP S/4HANA Data Migration Challenges
Strategies To Overcome SAP S/4HANA Data Migration ChallengesDipak Pimpale
 
SAP HANA SQL Data Warehousing (Sefan Linders)
SAP HANA SQL Data Warehousing (Sefan Linders)SAP HANA SQL Data Warehousing (Sefan Linders)
SAP HANA SQL Data Warehousing (Sefan Linders)Twan van den Broek
 
Integration of SAP HANA with Hadoop
Integration of SAP HANA with HadoopIntegration of SAP HANA with Hadoop
Integration of SAP HANA with HadoopRamkumar Rajendran
 
S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...
S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...
S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...lakshmi vara
 
What's Planned for SAP HANA SPS10
What's Planned for SAP HANA SPS10What's Planned for SAP HANA SPS10
What's Planned for SAP HANA SPS10SAP Technology
 
Empowering SAP HANA Customers and Use Cases
Empowering SAP HANA Customers and Use CasesEmpowering SAP HANA Customers and Use Cases
Empowering SAP HANA Customers and Use CasesthinkASG
 
SAP HANA SPS09 - HANA IM Services
SAP HANA SPS09 - HANA IM ServicesSAP HANA SPS09 - HANA IM Services
SAP HANA SPS09 - HANA IM ServicesSAP Technology
 
SAP_SLT_Guide_21122015.pdf
SAP_SLT_Guide_21122015.pdfSAP_SLT_Guide_21122015.pdf
SAP_SLT_Guide_21122015.pdfssuser17886a
 
SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...
SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...
SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...IBM
 
Sap deployment
Sap deploymentSap deployment
Sap deploymentAsha Panda
 
1310 success stories_and_lessons_learned_implementing_sap_hana_solutions
1310 success stories_and_lessons_learned_implementing_sap_hana_solutions1310 success stories_and_lessons_learned_implementing_sap_hana_solutions
1310 success stories_and_lessons_learned_implementing_sap_hana_solutionsBobby Shah
 
Digital economy with the speed of s4 hana
Digital economy with the speed of s4 hanaDigital economy with the speed of s4 hana
Digital economy with the speed of s4 hanaKyyba Inc.
 
HANA Demystified by DataMagnum
HANA Demystified by DataMagnumHANA Demystified by DataMagnum
HANA Demystified by DataMagnumPrasad Mavuduri
 

Ähnlich wie SAP Lambda Architecture Point of View (20)

209 hana-defining-capability-whitepaper
209 hana-defining-capability-whitepaper209 hana-defining-capability-whitepaper
209 hana-defining-capability-whitepaper
 
Analytics Products L2 public 2020-23 Black.pptx
Analytics Products L2 public 2020-23 Black.pptxAnalytics Products L2 public 2020-23 Black.pptx
Analytics Products L2 public 2020-23 Black.pptx
 
SAP HANA
SAP HANASAP HANA
SAP HANA
 
Strategies To Overcome SAP S/4HANA Data Migration Challenges
Strategies To Overcome SAP S/4HANA Data Migration ChallengesStrategies To Overcome SAP S/4HANA Data Migration Challenges
Strategies To Overcome SAP S/4HANA Data Migration Challenges
 
SAP HANA SQL Data Warehousing (Sefan Linders)
SAP HANA SQL Data Warehousing (Sefan Linders)SAP HANA SQL Data Warehousing (Sefan Linders)
SAP HANA SQL Data Warehousing (Sefan Linders)
 
Integration of SAP HANA with Hadoop
Integration of SAP HANA with HadoopIntegration of SAP HANA with Hadoop
Integration of SAP HANA with Hadoop
 
S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...
S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...
S4F01_EN_Col17 Financial Accounting in SAP S4HANA for SAP ERP FI Professional...
 
What's Planned for SAP HANA SPS10
What's Planned for SAP HANA SPS10What's Planned for SAP HANA SPS10
What's Planned for SAP HANA SPS10
 
Empowering SAP HANA Customers and Use Cases
Empowering SAP HANA Customers and Use CasesEmpowering SAP HANA Customers and Use Cases
Empowering SAP HANA Customers and Use Cases
 
SAP HANA SPS09 - HANA IM Services
SAP HANA SPS09 - HANA IM ServicesSAP HANA SPS09 - HANA IM Services
SAP HANA SPS09 - HANA IM Services
 
SAP_SLT_Guide_21122015.pdf
SAP_SLT_Guide_21122015.pdfSAP_SLT_Guide_21122015.pdf
SAP_SLT_Guide_21122015.pdf
 
SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...
SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...
SAP S/4HANA cloud editions or On Prem? Demystifying the options and cost bene...
 
Pol03262 usen
Pol03262 usenPol03262 usen
Pol03262 usen
 
Sap hana
Sap hanaSap hana
Sap hana
 
Sap deployment
Sap deploymentSap deployment
Sap deployment
 
SUSE Technical Webinar: Build HANA Apps in the Framework of the SAP and SUSE ...
SUSE Technical Webinar: Build HANA Apps in the Framework of the SAP and SUSE ...SUSE Technical Webinar: Build HANA Apps in the Framework of the SAP and SUSE ...
SUSE Technical Webinar: Build HANA Apps in the Framework of the SAP and SUSE ...
 
1310 success stories_and_lessons_learned_implementing_sap_hana_solutions
1310 success stories_and_lessons_learned_implementing_sap_hana_solutions1310 success stories_and_lessons_learned_implementing_sap_hana_solutions
1310 success stories_and_lessons_learned_implementing_sap_hana_solutions
 
Cool features 7.4
Cool features 7.4Cool features 7.4
Cool features 7.4
 
Digital economy with the speed of s4 hana
Digital economy with the speed of s4 hanaDigital economy with the speed of s4 hana
Digital economy with the speed of s4 hana
 
HANA Demystified by DataMagnum
HANA Demystified by DataMagnumHANA Demystified by DataMagnum
HANA Demystified by DataMagnum
 

Kürzlich hochgeladen

Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 

Kürzlich hochgeladen (20)

Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 

SAP Lambda Architecture Point of View

  • 1. SAP Lambda Architecture Point Of View SAP Digital Enterprise Platform Doug Martino January 2016 External
  • 3. SAP External | 3 Introduction New Architectures for Big Data Whether MVC for user interfaces, or Spine and Leaf for data centers, new architecture patterns in our industry act as sort of historical markers of the effectiveness and acceptance of new technologies. Practical techniques push the bounds resulting in a shift. Application of distributed storage and streaming capabilities such as Kafka and of course Hadoop are shifting Big Data architectures from a layer cake concept, or North/South oriented approach to one which can be thought of as an East/West architectural concept. Recently popular is Lambda Architecture, this article presents an SAP HANA based rendering of the Lambda Architecture. Lambda Architecture Before discussing how SAP products can be used in a Lambda fashion, a brief overview of Lambda is in order. Attribution must go to Nathan Marz for his work. Typically consisting of three components, a batch layer, a speed layer, and a serving layer, Lambda rethinks how data flows through an analytics system. Data flows left to right in a multi speed architecture; with all data stored in the batch layer and the speed layer simultaneously, while the serving layer is used to combine and analyze that data as required. Advantages are fault tolerance, scalability, while at the same time offering multiple latencies to supporting a a variety of performance requirements. With Lambda some business questions are answered using the speed layer, others answered by the batch layer, and still others answered by combing both. As algorithms in the speed layer decide which data to keep, answer immediate questions such as which ad to display or how much discount to offer, it also stores an immutable copy of the data into the batch layer. In addition to acting as a replay source, data in the batch layer is suited for use cases such as long tail analytics, machine learning, or propensity to ‘act’ types of endeavors. Fault tolerance, resilience, ultimate scale, and back testing are derived from having the immutable copy of the original data for replay through the downstream components. The serving layer may persist a copy of the data to be used for additional downstream use of the data such as dashboards or feeds to another system. Shown here are several open source options for the components which make up a Lambda Architecture.
  • 4. SAP External | 4 Traditional Lambda Architecture Lambda with SAP HANA Platform At SAP we are rapidly delivering value with our customers by employing these types of architecture patterns. In 2012 we started moving the HANA product towards the SAP Real-time data architecture where one already finds the notions of Lambda. Smart Data Streaming is the obvious choice for the real-time layer. Born on Wall Street, this component is capable of ingesting millions of rows per second into HANA. Complete with a rich set of API’s, stream and window compute constructs, and scale out capability, it can send the right data into HANA and all the data into HDFS as necessary. As the leading in-memory database platform, HANA meets the needs of the serving component of Lambda rather well. The data can be stored once, then transformed, aggregated, and calculated dynamically. Easy to understand in-memory compute engines such as graph, spatial, OLAP, and predictive analytics are available to manipulate the data where it is stored. Because the immutable data is kept in an append only columnar SQL database and the serving is done on the fly, our customer’s experience a valuable and unique combination of expressive, declarative programming capability on data which is managed in a SQL framework. This allows data access via everyday BI tools via standard ODBC or JDBC SQL as well as ODBO MDX. Based on Node.js, the HANA app server XS Advanced provides amongst other things noSQL access to the data via restful ODATA.
  • 5. SAP External | 5 Often times requirements change such that new algorithms or new derivations of the immutable data are required. Because of the dynamic nature of materialization employed by the HANA core architecture, these changes can be made almost instantly. This flexibility changes the logistics of replay, allowing for rapid experimentation and moves into a production environment. In the case that a schema needs to be extended HANA does have schema flexibility whereby columns may be added to an existing table structure, leaving any existing code intact. The amount of flexibility offered by HANA once the immutable data has been stored is a fundamental differentiator of HANA. This flexibility offers reduced time to value and increased data agility. While the HANA platform is working extremely well for real-time applications, Lambda is complete by incorporating Vora. From an architecture standpoint, there are several aspects to add to the discussion. First is the aspect of scalability.
  • 6. SAP External | 6 One of the beauties of the lambda is scaling each component dynamically and independently as latency requirements shift to meet changing business requirements. This notion of independent scaling allows appropriate resources allocation in a fit-for purpose model. Vora’s use of Hadoop moves the scale of the overall system up to multi petabyte range. More importantly, addressing long tail analytics and other big data compute problems are now possible. This leads to the next concept, which is one of throughput and algorithmic capability. Each of the real- time/batch/serving layers are best suited for a particular algorithm at a given throughout. All of them can run ‘out of band’ code; the question is at which level of concurrency and complexity. Rather than have JAVA JVM as the foundation of each, SAP have the appropriate engines with well-documented and supported extension capabilities. One may argue that open source provides the ultimate extension capability but only as a consuming organization is willing to merge code. In this case, SAP is providing high performance, fit for purpose engines to achieve the desired results more efficiently. This leaves us with the replay topic, an oft forgotten concept, especially in an operational batch context. Usually left for that – operations, it is often the case that re-running loads into a system to catch up with the real world, or to modify results based on new algorithms cause over provisioning of hardware in today's ETL and serving layers. By accounting for this capability up front, the opportunity to make transforms happen at the right time reduce the time and energy required to keep things in phase. HANA is a fantastic example of this given its powerful late materialization capability. Eventually elastic capabilities will further reduce replay workload stress by optimal allocation of compute resources with finer grain and lower latency. By employing lambda and other emerging big data architecture patterns, our customers remain well positioned as the state of the art continues forward.
  • 7. www.sap.com/contactsap © 2016 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see http://www.sap.com/corporate- en/legal/copyright/index.epx#trademark for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward- looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward- looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.