SAP Vora CodeJam

SAP VORA 1.4
July 2017
Puntis Jifroodian-Haghighi, SAP Vora Product Management
Jason Hinsperger, SAP Vora Product Management
Vitaliy Rudnytskiy, SAP Developer Relations

2INTERNAL© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
o Introduction to SAP Vora and Big Data
o Vora Installation and Configuration
o Tables and Views in Vora
o Break
o Different Data Sources
o Hierarchies
o Vora Graph Analysis
o Break
o Time Series, Document Store and Disk Engine
o Wrap-up
Agenda

For SAP, Big Data Expands the Customer Data Footprint
Enterprise Data Data Lakes
BW/4
HANA
ASE
IQ
HANA
D A T A L A K E

What is a Data Lake and Why the rise in popularity?
A Data Lake is a massive, easily accessible, centralized repository of data stored on commodity
hardware. Data is stored in its raw format and transformed when needed. Data Lakes can scale to
PB range, thus becoming the de facto standard for Big Data initiatives.
Scalable
Manage more data
than ever before
Store any data
Supports all data types
Flexible
Analyze and simulate
any business scenario
Dynamic
Deploy on premise or
in the Cloud

Hadoop: Cheaper, Faster, Effective Big Data Storage and Processing
• Highly scalable – handles terabytes
to petabytes and beyond
• Affordable – open source software,
runs on commodity machines
• Flexible – handles all varieties of data,
not just structured data
• Tackles the data challenges that
drive modern business – Yahoo!,
Netflix, Facebook all use Hadoop to
run core business

Challenges with Big Data
* Source: Gartner Big Data Adoption Survey insights November 2016 https://www.gartner.com/webinar/3451618
Achieving Value, Skills, Capabilities, Governance

Unlock Business Potential from Your Big Data
Insights from
one single solution
In-memory distributed computing
engines: Relational, Time Series,
Graph, JSON/Doc
Disk-to-memory accelerator
Enterprise-ready
Production-ready,
integrated solution
Support, Versioning and
Compatibility between different
Hadoop components
Seamless integration with
SAP HANA
Easier to use
Intuitive web interface
One SQL entry point
Connect with familiar tools

DW/Data Tiering
• Economically maintain context
data without compromising on
performance
• Immediate results from
petabytes of contextual data
without compromising core
business systems
IoT
• Immediately sense and respond to
data streams with Vora’s
embeddable design
• Off-load processing to edge
devices and simplify data
synchronization with gateways
and data warehouses
Data Lineage &
Compliance
• Query archived data without
losing data lineage & control
• Deliver timely, accurate
compliance reporting
• Perform complete audit trail
analysis from operational to
archive data
Cloud
• Analysis of large volumes of
external data without building
on-premise infrastructure
Data Lake
• Faster analytics with
compiled query for
distributed processing
• Better insights with drill-
down root-cause analysis
SAP Vora Usage Patterns

Use Case Challenge Why SAP
HANA Vora?
• Significant compute capacity
required for valuations
• Store larger volumes of data in
Hadoop and leverage Spark + Vora
to deliver massive parallel compute
power + enterprise-grade analytics
Risk
Management
Fraud
Detection
• Early detection of rogue/illicit trading
• Leverage Vora to correlate
seemingly unrelated transactions to
identify fraudulent activity
Broker/Trade
Compliance
• Detect fraudulent transaction earlier
to improve accuracy and reduce
costs
• Link trading and Financial (ERP/S/4)
transactions with unstructured
content (emails/instant
messaging/sentiment data)
Anti-Money
Laundering
• Detect illegal activities faster and
with improved accuracy
• Ingest, store and process large
amounts of data and leverage
machine learning algorithms to
detect money laundering activities
Customer 360˚
• Reduce Customer Churn
• Grow the business with targeted
marketing campaigns
• Better understand your customer
behaviors and needs (sentiment
analysis) to proactively make
relevant offers and reduce churn
SAP Vora for Financial Services/Capital Markets

SAP Vora Telco/Network capacity planning/Real-time bandwidth allocation
 Network Capacity planning – By analyzing Call
Detail Records (CDRs) and network loads, telcos
can plan infrastructure expansion with greater
precision
 Cellphone service improvement -Ability to
analyze, understand and fix instances of poor
cellular service e.g. dropped calls/poor audio
 Targeted network maintenance/upgrades –
Analysis of how cable network congestion affects
churn, and where exactly network upgrades produce
the most incremental revenue
 Real-time bandwidth allocation: With Real-time
packet inspection operators can steer traffic and
optimize network quality of service (QoS) in real time
in an attempt to maintain the best service quality

SAP Vora Oil & Gas Environmental Safety/Oil Production/Predictive Maintenance
 Environmental Safety: Analysis of anomalies in
drilling can be identified in real time. Well problems
can be detected before they become serious and
drills can be shut down proactively to prevent
environmental risks
 Oil Production Forecasting: Analysis of seismic,
drilling and production data to enhance oil
extraction from existing wells and forecast oil
production
 Identification of potential drilling errors
/equipment failures, by analyzing sensor data from
equipment (drill heads, down hole sensors, etc.) as
well as geological data. Understand what
equipment works best in each environment
 Predictive Analysis : Identification of events or
patterns that could indicate an imminent security
threat or cyber- terrorist act in order to keep their
personnel, property and equipment safe

SAP Vora Retail/Consumer Buying Patterns/Dynamic Pricing
 360° Customer Insight: Improve customer
satisfaction and increase sales opportunities by
integrating all relevant customer data across online
transactions, POS transactions, social media, and
customer service interactions into one single view
 Upsell/cross-sell recommendations: Increase
online purchases by recommending relevant
products and promotions in real time. Retailers can
recommend products based on what other similar
customers have bought
 Click Stream Analysis: Clickstream analysis helps
retailers to better understand how consumers make
online purchase decisions which in turn helps them
to optimize web pages/offers to increase
conversion, resulting in lower cart abandonment

SAP Vora - Manufacturing
Potential Use Cases
 Predictive Maintenance – Minimize Non-Productive Time
(NPT) by monitoring equipment or product utilization in a
live environment to identify patterns that indicate imminent
failure.
 Assembly Line Quality Assurance: Take measurements
of work-in-progress products to find manufacturing defects
as early as possible, while also identifying any potential
process or design flaws. Analysis of
 Real-time Parts Flow Monitoring: Attaching sensors to all
parts in the production process and tracking them in real
time enables manufacturers to have a real-time view to
their production process.
 Product Configuration Planning: Product configuration
planning helps accelerate production by offering fast
delivery times for the manufacture of millions of different
product configurations.

3xV of Big Data

Hadoop Overview

SAP Vora in the Hadoop Ecosystem

Apache Spark is a fast and general engine for large-scale data processing
Can run along with, or independent from Hadoop
Can work against data from many sources – HDFS, local files, Amazon S3, etc…
Supports Java, Scala or Python based applications
Works on data in an in-memory fashion
– 10-100 times faster than Hive on Map/Reduce
Supports SQL access
Includes a Streaming engine
Apache Spark

What does SAP Vora provide which Spark does not?
Integration with Enterprise data
– Move processing closer to the data by extending Spark to allow the computation of full logical plans at the datasource (whether its HANA
or the Vora Engine)
– The result is a performance increase (but dependent on the actual query)
Bidirectional virtual data access from HANA to Hadoop/Spark extending beyond SDA capabilities
– No data duplication
Enterprise Analytics additions to SparkSQL
– Hierarchies, currency conversion
Unified access layer built on SQL allowing integrated analysis of disparate datasets
Simplified data modeling on HDFS/S3 data sets using web based modeler
SAP Vora Complementing Spark Framework

Distributed Computing Solution for the Digital Enterprise
HadoopFiles Files Files
Vora
Spark
Vora
Spark
Vora
Spark
…
Distributed Transaction Log
Disk-to-Memory Accelerator
Data Modeler
Relational Time Series Graph Doc Store
SAP VORA
Data Science, Predictive, Business Intelligence, Visualization Apps
Distributed computing cluster
Spark
Hadoop
Other Apps
In-Memory
Store
SAP
HANA
Platform
O P T I O N AL

Key Capabilities of SAP VORA
Vora Vora Vora
Vora Vora Vora
Vora Vora Vora
Native „Database“
in Hadoop
Multiple Engines
Relational
(OLAP)
In-
Memor
Time
Series
Doc Store
Graphs
Intuitive Tools Tight HANA Integration
0.1sec
∞
HANA
Hadoop

• Full support for Kerberos (AD, MIT) enabled Hadoop
landscapes
• Encrypted HDFS support
• Access Hadoop transparent encryption zones
• Consul and Nomad TLS (Transport Layer Security) & ACL (Access
Control Lists) support for Vora services
• Secure access to various Vora components ( engines, Tools UI,
catalog)
Security

:To administer and monitor the Hadoop landscapeCluster Management Tools
Cloudera Manager MapR Control System Ambari

Vora Nodes
Vora Spark Extension
SAP VORA
Transaction
Coordinator
Metadata Scheduler Discovery Landscape
DLog S3 / Swift / HDFS/ORC/Parquet
Control
Nodes
(1..few)
Compute
Nodes
(1..many)
Relational
Documents Graph
Time Series
Disc
Persistent
Storage
Relational
Documents Graph
Time Series
Disc

Vora Manager UI http://<manager_node>:19000
Shows the status of all services
▫ Start/stop all services
▫ Configuration management:
 specify node assignments
 change parameters
Consul
▫ Vora Discovery Service UI http://<discovery_server_node>:8500/ui
▫ Know where the services are
Nomad
▫ Process scheduler and resource manager
▫ Start/stop/restart of Vora services -- If a service fails, Nomad will
automatically keep trying to restart it until it succeeds
▫ Manage node assignment
Vora Manager Components

Vora Manager UI: Home Page

Vora Manager UI: User Management

Vora Manager UI: Nodes

Vora Manager UI: Services

SAP Vora Nodes: Control Nodes
SAP VORA
Transaction
Coordinator
Metadata Scheduler Discovery Landscape
Control
Nodes
(1..few)

• Transaction Coordinator
• manages user transactions
• Transaction Broker
• enforces consistent (meta)data
modifications
• Landscape Manager
• Controls data partitioning and
placement across different engines
• Lock Manager
• Provides a distributed read-write lock
mechanism for concurrent load
statements to avoid loading the same
partition multiple times
• A driver for query execution with user
session semantics
Control Nodes

Query Processing
Query
Processor
Catalog
Landscape
Manager
Host
Assignment
Scheduler
User Query
Vora Engines (graph, doc, series, disk)
Result Set

Vora Nodes: Compute Nodes
SAP VORA
Compute
Nodes
(1..many)
Relational
Documents Graph
Time Series
Disc
Relational
Documents Graph
Time Series
Disc

 Relational Engine – allows you to frame your data as relational entities and communicate with them using SQL
 Graph engine – SAP VORA embeds an in-memory graph database for real-time graph analysis. The primary focus is on complex read-only analytical queries on
very large graphs.
 Time Series – SAP VORA provides a highly-distributed time series analysis engine which supports storing and analyzing time series data. By enabling efficient
(memory and speed) time series compression and supporting features like standard aggregation, granularization, and advanced analysis; SAP VORA allows you to join
the relational data with series data to build efficient SQL models in Hadoop and other Big Data environments
 Document Store – SAP VORA introduces NoSQL features like storing JSON documents using the new Document Store as part of the SAP VORA 1.3 release. The
new DocStore supports schema-less tables, allowing you to flexibly add or remove fields from any documents and helps scale horizontally
 Disk to Memory Accelerator – SAP VORA provides relational capabilities without having to load all the data into memory in those cases where it will not fit.
Vora Analysis Engines
Distributed Transaction Log
Disk-to-Memory Accelerator
Data Modeler
Relational Time Series Graph Doc Store
SAP VORA
Spark
Hadoop

Vora Nodes: Persistent Storage
SAP VORA
DLog S3 / Swift / HDFS/ORC/Parquet
Persistent
Storage

Persistent Storage: HDFS + CATALOG + DLOG
Vora Catalog
(Metadata)
Vora
DLOG
Spark
Context
REGISTER ALL TABLES USING com.sap.spark.vora
REGISTER TABLE <name> USING com.sap.spark.vora
SHOW TABLES USING com.sap.spark.vora
Spark parameter: spark.sap.autoregister com.sap.spark.vora
Persistent
Volatile
(Spark is session-based)
• Shows tables in local Spark Context
• After restart (e.g. of spark-shell) Vora table metadata
needs to be reloaded into local Spark Catalog via
REGISTER
Spark Catalog
SHOW TABLES

Take a few minutes and explore the Vora manager, configuration components and various settings
available.
 SAP Vora Manager: http://<IP_ADDRESS>:19000
Check It Out!

On-Premise
– Developer Edition on Premise: http://developers.sap.com
– Install Official Vora Release on SAP SMP
Cloud Based
– Vora on CAL
– Amazon AWS with “one-click” – developer and production editions (AWS marketplace)
Vora Testdrives
– 3hr testdrives - http://testdrive.saphanavora.com/
▫ Retail
▫ Telco
▫ Time series
Vora Installation Options

– Install Official Vora Release
– Automated scripts for you own cluster (Ansible)
– Monsoon
Options to start with Vora
On premise and in the cloud developer editions:
http://www.sap.com/developer/topics/vora.html#freetrial

Set up and Start developer editions
https://www.sap.com/developer/tutorial-navigator.how-to.html?tag=products:data-management/sap-hana-vora

• Pre-defined business scenario--sample data
included
• Step-by-step instructions
• Includes all solution components (e.g. Vora +
Lumira)
• 100% free
• Time bound
Test Drive Lab vs. Developer Edition
http://testdrive.saphanavora.com/

Features
• Setup fully functional and configured SAP Vora cluster with few clicks. Available in most of the
AWS regions.
• SAP Vora Console to manage and scale the cluster
• Optimal 4 node cluster based on
• Centos 7.2
• Vora 1.3.61 (GA)
• Apache Ambari 2.2.1.0
• Spark 1.6.1
• Hadoop Distribution HDP 2.4.2
• Zeppelin 0.6.0
SAP Vora Developer Edition on AWS

Prerequisites
Amazon Web Services account
Virtual Private Cloud (VPC) and Security Group as virtual firewall
Steps
Go to Vora sign up page to launch an instance from an Amazon Machine image (AMI)
Use SAP Vora console to setup cluster, add nodes and configure
Use console to view and manager the cluster
Get your SAP Vora Dev edition in AWS

How can I communicate with Vora?
Spark Adapter
SAP HANA
Thrift Server
Vora
Spark
Vora JDBC /
ODBC
Vora
Tools
Type
• Native Spark
• JDBC
• Graphical
• HANA

Vora Consumption
Consumption from HANA is possible through
 SDA with “Spark-Adapter” (since SAP HANA SPS 10)
 SDA with Thrift Server (Since SAP HANA SPS 7)
 Direct Connectivity (End of 2016)
JDBC / ODBC Access
 Spark SQL is exposed through the Thriftserver (SAP VORA
Extensions included and using SapSQLContext)
Data visualization tools
 Apache Zeppelin (or Jupyter) via SAP Spark extensions
 SAP Lumira Data Analysis via JDBC channel
 SAP VORA Data Modelling via JDBC channel
Native language bindings
 for programmatic data access
 Available in Scala, Java, Python, R
How can I communicate with Vora?
Consumption of SAP VORA
Spark
Adapter
SAP HANA
Thrift
Server
Vora
Spark-Driver
SAP Spark Extensions
Vora
ODBC
Vora
Tools

– Vora Tools UI, http://<tools_node>:9225
– SQL Editor
– Data Browser
– Modeler Perspective
– User Management
Vora Tools

Zeppelin, Spark Shell and Jupyter Notebook
Apache Zeppelin
Jupyter Notebook
Spark Shell
• Spark-Shell with Vora
• $ source /etc/vora/vora-env.sh
• $ $VORA_SPARK_HOME/bin/start-spark-shell.sh
• Also possible: spark-submit; pyspark; other tools that can connect to the Vora
Thriftserver

Lumira
– Graphical Data Exploration
SAP Lumira
SAP Lumira, Zeppelin:
• Installation and Administration
Guide (do not install Zeppelin via Ambari)
Jupyter:
• https://blogs.sap.com/2016/01/21/visualizing-data-with-jupyter/

SAP HANA
– Virtual Table via SDA
– Remote Source
▫ VoraODBC Connector
▫ Spark Connector
SAP HANA (via Smart Data Access, SDA)

Pluggable API to access structured
data through Spark SQL
Data filtering and column pruning can
be pushed down to the data sources
in many cases (depends really on the
source capability)
Advantages:
• Less data in Spark, Less disk IO
• Less to do in Spark
• Less memory consumption
• Any Spark language can leverage
Data Sources API
Spark Data Sources API: Quick Refresher
Spark SQL
Spark Core Engine
Data Sources
MLlib
Spark
Streaming
GraphX
(graph)
CSV HANA

• Extends Spark Data Source to allow the creation of
relations in Vora
• The tables stored in SAP VORA are exposed to Spark as
standard Spark DataFrames
• To query the relations use standard Spark SQL or Vora
SQL
• Vora data source analyzes Spark SQL queries for
opportunities to push the execution of parts or all of the
query to SAP VORA
• Vora data source, a Spark SQL query can be executed on
multiple SAP VORA engines concurrently
• the execution result returned back to the Spark runtime as
an RDD (resilient distributed data set)
SAP VORA Data Source (1)

SAP VORA provides the following data sources:
• com.sap.spark.hana for the SAP HANA data source
• com.sap.spark.engines for the specific SAP VORA engines (relational,
graph, time series, document store, and disk)
• com.sap.spark.vora for the SAP VORA data source (deprecated)
SAP VORA Data Source (2)

SQL Code:
CREATE TABLE USERS (user_id string, age
integer, gender string, occupation string,
zip_code string)
USING com.sap.spark.engines.relational
OPTIONS (
files
"/path/to/USERS1.csv,/path/to/USERS2.csv”
)
Working with Tables and Views in Vora
Creating Tables
• Options: storagebackend, format, csvdelimiter, csvquote, null,
csvskip, and datetimeformat.
• A Create TABLE statement, registers the table in the SparkSqlContext and
creates a table in the SAP VORA engine.

• Create table if table exists in Vora catalogue
• Create table if not exist
• Create table without data
• Create table without scheme
Create Table Conditions

 APPEND TABLE testTableName
OPTIONS (files
"path1/to/file/file1.csv,path2/to/file/file
2.csv”)
 APPEND TABLE testTableName
OPTIONS (files "path1/*”)
Working with Tables and Views in Vora
Register, Append and Drop
 DROP TABLE/VIEW testTableName
 DROP TABLE/VIEW IF EXISTS tableName
 DROP TABLE/VIEW tableName CASCADE
Use wild cards to read from hdfs files in a folder:
OPTIONS ( ...
files "/dir1/*,/dir2/*"
... )

Working with Tables and Views in SAP VORA
Creating tables (2)

 Listing Tables and Views:
Listing and loading tables and views in SAP VORA
REGISTER ALL TABLES USING
com.sap.spark.vora
REGISTER TABLE TABLE1
USING com.sap.spark.vora
IGNORING CONFLICTS
SHOW TABLES
OPTIONS (…)
// show only tables registered in the
Spark catalog
SHOW TABLES
 Loading Tables from SAP VORA into Spark:

Views:
 SQL
CREATE VIEW MyView
AS SELECT * FROM Table1
OPTIONS ( …)
 Dimensions
CREATE DIMENSION VIEW MyDimensionView
OPTIONS (…)
Persisted views
 Cubes
CREATE CUBE VIEW MyCubeView
OPTIONS (…)
 DROP VIEW MyView
OPTIONS (…)
 DESCRIBE TABLE MyView USING com.sap.spark.vora

© 2017 SAP SE or an SAP affiliate company. All rights reserved. 65PUBLIC
1. Financial institutions
2. Products
3. Complaints

© 2017 SAP SE or an SAP affiliate company. All rights reserved. 66PUBLIC
Complaints.csv
FinancialInstitutions.csv
Products.csv

Take 15 minutes and get familiar with Apache Zeppelin, and execute the commands in the following
notebooks:
– 0_DATA – create base tables
– Tables and Views – create views, cubes and dimensions on the base tables
 https://www.sap.com/developer/tutorials/vora-ova-zeppelin0.html
 Apache Zeppelin: http://<IP_ADDRESS>:9099
Check It Out!

Vora in-memory
Review – What did we do?
CREATE TABLE…
…
Thrift Server
Spark
SELECT * FROM
TABLE…
Data
Data File

SAP VORA 1.4
Different Data Sources

CREATE TABLE Users(id integer, name string)
OPTIONS (
files "/S3_BUCKET/data.csv",
csvdelimiter "|",
storagebackend "s3",
s3accesskeyid "S3_KEY_ID",
s3secretaccesskey "S3_KEY_SECRET",
s3endpoint "S3_ENDPOINT",
s3region "S3_REGION" )
Loading Different Data Types
Loading data from Amazon S3

OPTIONS (
tablename "Users",
files "/ORC_Files/data.orc/*",
format "orc"
)
OPTIONS (
tablename "Users",
files "/Parquet_Files/data.parquet/*",
format "parquet"
)
Loading Different Data Types
Loading data from ORC and Parquet file

Individual tables via Vora
All tables via Scala
 import com.sap.spark.engines.client.EngineClient
 EngineClient.getOrCreate().reloadTables()
Reloading tables into Vora engines after restart

 SAP HANA DATA SOURCE
CREATE TABLE $tableName
USING com.sap.spark.hana
OPTIONS
(
tablepath "$tableName”,
dbschema "$dbSchema”,
host "$host”,
Instance "$instance”,
user "$user”
passwd "$passwd”
)
Loading Different Data Types: SAP VORA – SAP HANA Connection
Loading data from/to SAP HANA
SAP HANA
In-Memory
Store
Spark SQL
Smart Data Access Spark Extension
Spark SparkSpark

Writing Data into HANA

SHOW TABLES USING com.sap.spark.hana
OPTIONS (
host "$host”,
instance "$instance”,
user "$user”,
passwd "$passwd”,
dbschema "$dbSchema”,
tablePattern "$pattern”
)
Listing Tables in SAP HANA

REGISTER TABLE tablename
OPTIONS (
host "$host”,
user "$user”,
passwd "$passwd”,
dbschema "$dbSchema”) [IGNORING CONFLICTS]
REGISTER ALL TABLES USING com.sap.spark.hana
OPTIONS (…)
Register HANA tables to current Spark Context

DROP TABLE testTableName
DROP TABLE IF EXISTS testTableName
DROP TABLE testTableName CASCADE
DESCRIBE TABLE tablename
OPTIONS (
host "$host”,
user "$user”,
passwd "$passwd”,
dbschema "$dbSchema”)
Drop/Expose metadata

Pushing down the Hana UDF’s

Take 10 minutes and go through the tutorial to connect Vora to HANA and to read different file
formats.
https://www.sap.com/developer/tutorials/vora-ova-zeppelin6.html
https://www.sap.com/developer/tutorials/vora-ova-hana-datasource.html
 SAP Vora Tools: http://<IP_ADDRESS>:9225
Check It Out!

• Relational column based store
• You can create, register, and query disk engine tables in Spark SQL
statements in the same way as with other data sources (for example,
SAP VORA, SAP HANA, and native Spark data sources). The syntax
and options of the CREATE TABLE statement are compatible with the
SAP VORA data source.
• Uses ‘com.sap.spark.engines.disk’
• REGISTER ALL TABLES USING com.sap.spark.engines.disk
Vora Disk to Memory Accelarator

Take 10 minutes and work through the doc store notebook in Apache Zeppelin
Check It Out!

Working with Hierarchies in SAP VORA
What are hierarchies?
Clothes
Men Women
Evening
Gowns
SkirtsShirts SuitsShorts
JacketsSlacks
Dresses Blouses
Sun
Dresses

Parent-Child Hierarchy (aka Adjacency-List)
Clothes
Men Women
Evening
Gowns
JacketsSlacks
Dresse
s
Blouse
s
Sun
Dresses
1
32
4 5 6 7 8 9
10 11 12 13

• ORDER SIBLINGS BY: is relevant for some UDFs, such as IS_PRECEDING and IS_FOLLOWING
• START WHERE
• SET
Create Parent-Child hierarchy from the clothes table
Clothes
Men Women
Evening
Gowns
JacketsSlacks
Dresse
s
Blouse
s
Sun
Dresses
1
32
4 5 6 7 8 9
10 11 12 13

Parent-Child Hierarchies

h_src(level 1, level 2, level 3, level 4)
Level-Based Hierarchies (aka flattened hierarchies)
Clothes
Men Women
Evening
Gowns
JacketsSlacks
Dresse
s
Blouse
s
Sun
Dresses
1
32
4 5 6 7 8 9
10 11 12 13

• WITH LEVELS (col1, col2, col3, col4)
• MATCH PATH: Determines the way identical nodes are handled across the same and across different
columns.
• ORDER SIBLINGS BY
• SET
Level Hierarchies – Create statement

Working with Hierarchies in SAP VORA
Hierarchy UDFs
UDF
level(u)
is_root(u)
is_descendant(u,v)
is_descendant_or_self(u,v)
is_ancestor(u,v)
is_ancestor_or_self(u,v)
is_parent(u,v)
is_child(u,v)
is_sibling(u,v)
is_following(u,v)
is_preceding(u,v)
node(node)

Using UDF’s with Hierarchies - example

Using UDF’s with Hierarchies

Take 10 minutes and work through the hierarchies notebook in Apache Zeppelin
Check It Out!

SAP VORA 1.4
Time Series Engine

Time Series and Time Series Analysis
-30
-25
-20
-15
-10
-5
0
5
Temperature °C
Halifax Waterloo
Time Series: Sequence of data points recorded over time, may
occur as equidistant / non-equidistant
Detect and correct errors / anomalies: Outlier Detection,
Missing Value Replacement, Editing
Standard aggregation: SUM, MIN, MAX, Select, Join,
Grouping across Series (e.g.: group by province)
Granularization Support: Hourly to Daily measurements
Series Analysis: Smoothing, Binning, Correlation

Time Series Data Analysis across big data
Efficiently analyze time series data in distributed
environments
 Interactive access to standard time series
analysis functions using the well-known SQL
language
 Efficient compression allowing analysis of
more data using less memory
 Build time series models visually using Vora
Data Modeler
Trend | Cyclical | Seasonal | Random | Exception

• Identified series column
• Know what the series period is
• Think about compression strategy
eg. Max error bound for compression
• Define partition function
• Define partition scheme
Creating a Series Table - Prerequisites

CREATE TABLE "name" ( "time" TIMESTAMP,
"value1" INTEGER DEFAULT NULL,
"value2" DOUBLE )
SERIES ( PERIOD FOR SERIES "time"
START TIMESTAMP '2010-01-01 00:00:00'
END TIMESTAMP '2010-12-31 00:00:00'
EQUIDISTANT INCREMENT BY 1 HOUR )
PARTITION BY PS1("time")
Creating a Series Table - Examples

CREATE PARTITION FUNCTION PF1(C TIMESTAMP)
AS RANGE
BOUNDARIES(TIMESTAMP '2010-04-01 09:00:00.0000',
TIMESTAMP '2010-09-01 09:00:00.0000');
CREATE PARTITION SCHEME PS1 USING PF1;
Creating a Series Table – Partitions Example

ALTER TABLE name ADD DATASOURCE (
ts,
j thousands separated by ',',
d thousands separated by ',' decimal separated by '.'
)
'file_path_and_name'
delimited by ';'
string delimited by ' '
thousands separated by ','
decimal separated by '.';
LOAD TABLE name;
Add data to TS Table: Importing Data

The definition is stored in the catalog, so whenever the data is loaded into memory
the compression is applied.
``LOAD TABLE name`` using com.sap.spark.engines;
Load Table

CREATE TABLE national_grid_demand (
ts TIMESTAMP, ND double, I014_ND double, TSD double, I014_TSD double,
England_Wales_Demand double, Embedded_Wind_Generation double,
Embedded_Wind_Capacity integer, Embedded_Solar_Generation double,
Embedded_Solar_Capacity integer )
SERIES ( PERIOD FOR SERIES ts
START TIMESTAMP '2015-01-01 00:00:00' END TIMESTAMP '2017-01-01 00:00:00'
EQUIDISTANT INCREMENT BY 30 MINUTE DEFAULT
COMPRESSION use (APCA error 3.0 percent)
COMPRESSION ON (Embedded_Wind_Generation) use (SDT error 4.0 percent)
COMPRESSION ON (Embedded_Solar_Generation) use (SDT error 5.0 percent) )
PARTITION BY PS1( ts )
USING com.sap.spark.engines
OPTIONS (files "/user/<userid>/DemandData_2015_2.csv",
csvskip "1",
csvdelimiter ";",
storagebackend "hdfs")
Add date to TS Table: Create and Include Data

SELECT ts, ND FROM national_grid_demand
WHERE PERIOD AS OF TIMESTAMP '2015-09-01 00:00:00‘
SELECT ts, ND FROM national_grid_demand
WHERE PERIOD BETWEEN TIMESTAMP '2015-08-31 00:00:00' AND TIMESTAMP '2015-
09-06 00:00:00'
Querying Series Data - Examples

Standard aggregations
 SUM | AVG | MIN | MAX | COUNT
Trend
 Calculates the non-stationary behavior of the column using exponential smoothing
Median
 Median value for a time series column
Mode
 Modal value for a time series column
Querying Series Data – Column Functions

SELECT median(Embedded_Wind_Generation), median(Embedded_Wind_Capacity)
FROM national_grid_demand
09-01 00:00:00';
SELECT TREND(Embedded_Wind_Capacity), TREND(Embedded_Solar_Capacity)
FROM national_grid_demand
01-01 00:00:00';

Auto Correlation
• Calculates correlation coefficient for a given column and lag
• Allows you to look at a single column and find if there is a repeating pattern between the values of a column
Cross Correlation
• Calculates correlation coefficient for a 2 columns and a given lag
• Find if there is a correlation between multiple columns of a table
Histogram
• Calculate the frequency distribution of the values for the specified column
• Shows what is the distribution of your values in your time series over a time period
Granulize
 Returns a new series based on an existing series by changing the interval between adjacent time
stamps
Querying Series Data – Table Functions

SELECT * FROM HISTOGRAM (
SERIES national_grid_demand, 10, DESCRIPTOR( Embedded_Wind_Generation ) ) HIST;
SELECT * FROM AUTO_CORR (
SERIES national_grid_demand, 48, DESCRIPTOR( ND ) ) auto_corr_res;
SELECT ts, Embedded_Wind_Generation, Embedded_Wind_Capacity, Embedded_Solar_Generation,
Embedded_Solar_Capacity
FROM GRANULIZE(
SERIES national_grid_demand,
24 HOUR,
ROUND_HALF_UP,
SUM => DESCRIPTOR( Embedded_Wind_Generation, Embedded_Solar_Generation ),
AVG => DESCRIPTOR( Embedded_Wind_Capacity, Embedded_Solar_Capacity) )
WHERE PERIOD BETWEEN TIMESTAMP '2015-08-01 12:00:00' AND TIMESTAMP '2015-08-03 12:00:00';

SELECT * FROM
(``TS select statement 1`` USING … ) as T1,
(RT select statement USING … ) as T2,
WHERE
T1.v1 = T2.v1
Join between TS table and Relational table

Take 15 minutes and work through the time series notebook in Apache Zeppelin
Check It Out!

SAP VORA 1.4
Document Store Engine

Schemaless
 Very flexible
 Add/Remove fields to any document
Horizontally scalable (scale-out)
 Good for big data processing
Very low latency for key lookups
Easy to use (SQL syntax)
Document Store Overview

What does it store?
{
name: „Joe“,
age: 25,
hobbies: [„soccer“, „swimming“],
address: {
street: „4 Pennsylvania Plaza“,
city: „New York“
}
}
Field: Value
Field: Value
Field: Value
Field: Value

Documents are stored in Collections
{
name: „Jim“,
age: 25,
address: {
street: „4
Pennsylvania Plaza“,
citity: „New York“
}
}
{
name: „Jane“,
age: 25,
address: {
street: „4
citity: „New York“
}
}
{
name: „Joe“,
age: 25,
address: {
street: „4
city: „New York“
}
}

Schemaless – Missing Fields
{
firstName:"John",
lastName:"Smith",
age:22,
address: {
streetAddress: "21 2nd Street",
city: "New York",
state: "NY",
postalCode: "10021" }
}
{
firstName:”George",
lastName:”Brown",
licensePlate:”1ABC234”
address: {
streetAddress: "69 1st Street",
city: "Los Angeles",
state: ”CA” }
}

Schemaless – Different Types
{
firstName:"John",
lastName:"Smith",
age:22,
address: {
streetAddress: "21 2nd Street",
city: "New York",
state: "NY",
postalCode: "10A021" }
}
{
lastName:”Brown",
age:22,
address: {
streetAddress: "69 1st Street",
city: "Los Angeles",
state: ”CA” ,
postalCode: 34707 }
}
NUMBERSTRING

Schemaless – Different Types
{
firstName:"John",
lastName:"Smith",
age:22,
address: { ... },
phoneNumber: [
{ type: "home", number: "212 555-1234” },
{ type: "fax", number: "646 555-4567” }
]
}
{
lastName:”Brown",
age:22,
address: { ... },
phoneNumber : {
countryCode:90,
number:” 212 666-1234”}
}
OBJECTARRAY

How to use Doc. Store: Create Collection
CREATE COLLECTION <collection_identifier>
[PARTITION BY <partition_scheme_identifier>]
USING com.sap.spark.engines OPTIONS (<list_of_options>);
eg.
CREATE COLLECTION T
PARTITION BY PS2("state")
USING com.sap.spark.engines
OPTIONS (files 'path/to/file/to/load/some.json‘,
storagebackend "hdfs" );

Partitioning
CREATE PARTITION FUNCTION <partition_function_identifier>(<list_of_fields>)
AS HASH(<list_of_fields>) MIN PARTITIONS <num> MAX PARTITIONS <num>
USING com.sap.spark.engines;
CREATE PARTITION SCHEME <partition_scheme_identifier>
USING <partition_function_identifier> USING com.sap.spark.engines;
eg.
CREATE PARTITION FUNCTION PF("_id") AS HASH("_id")
MIN PARTITIONS 3 MAX PARTITIONS 3
USING com.sap.spark.engines;
CREATE PARTITION SCHEME PS
USING PF USING com.sap.spark.engines;

SQL Queries – Result Sets
SELECT
 Returns a relational result
SELECT {}
 Returns a JSON document
Use Dot Notation to access nested fields in a document
eg. {
"author": {
"address": {
"street": "..."
},
...
},
...
}
–Nested blocks accessed via array ‘[ ]’ operator
SELECT author.addresses[arraynum].street FROM <collection>

Vora Doc Store
Queries
 Standard SQL support
 SELECT
– Returns a relational result
 SELECT { }
– Returns a JSON document
 IS MISSING – look for missing fields similar to IS NULL
 Standard SQL support – sorting, subselect, alias, joins
group by
sum, avg, min, max, count
upper, lower, modulus

Select – Different Types
Always STRINGIFY in case of type mismatch
 Even objects and arrays
SELECT s.firstName,
s.address.postalCode
FROM Students s
firstName postalCode
”John” ”10A021”
”George” ”34707”

SQL Queries - Examples
SELECT {"ct": "city"} FROM <collection>
WHERE employeecollection.address.street = ‘Maple’
SELECT city as ct FROM <collection>
WHERE employeecollection.address.street = ‘Maple’
SELECT * FROM <collection>
WHERE author.address IS MISSING using com.sap.spark.engines;
SELECT * FROM <collection>
WHERE author.address[1].state=‘NY’

SQL Queries – Expressions and Aggregates
Standard Aggregate Functions are Supported
 count, min, max, avg, sum, etc…
GROUP BY
 Behaves as expected for a relational result
 For a JSON result, a new document is created where the field of the GROUP BY clause becomes the id of the
resulting document
 The _id field of the new documents is an object of grouped field names:
SELECT { _id: _id, aggr: avg(cost) } FROM <collection>
GROUP BY productgroup, productname using com.sap.spark.engines;
_id: {productgroup:value, productname:value}

Take 10 minutes and work through the doc store notebook in Apache Zeppelin
Check It Out!

Graph Databases
• Relationships between entities are of key interest
• Network view on complex structures
• Complex search patterns, e.g. subgraph search and link predictions
Application Areas
• Social networks
• Business networks (Users&Products, Learners&Contents)
• Knowledge graphs (Entities and concepts/relationships around them: Apple)
• Recommendation systems
• Org-/project structures (hierarchies / DAGs)
• Supply chain management
Graph Databases

Building Knowledge Graph
Build and infer relations between data points
Machine LearningMachine Teaching
hasLearningItem
SAP
hasOrgUnit
hasOrgUnit
Puntis
Product Manager
Balaji
Product Manager
Machine Teaching

SAP VORA Graph: Scalable Graph Analytics
Graph Analytics for
Enterprise Applications
Expressive, declarative graph query
language based on SQL and graph
pattern matching
High Performance for
Real-Time Applications
Native in-memory graph store and
light-weight query engine for super
fast query execution
Flexible Data Import from
Various Sources
Easy mapping from relational
tables to graphs. Direct loading
from files, HDFS, VORA catalog
Platform and
Application Integration
Part of VORA, connect to HANA
Distributed Engine
for Scale-Out Scenarios
Analyze billion-node graphs using
distributed version of the graph
store and query engine

Property Graph Model
Nodes have a type, a unique numeric ID
(64-bit integer > 0), and a set of
primitive-valued properties
Edges have a type and connect two
nodes
Directed and undirected graphs are
supported
Node properties have a type and a
primitive-typed value (nullable)
Supported primitive datatypes:
 64-bit signed integers
 variable-length strings
 double-precision floats
Graphs have no fixed schema
Properties can be indexed

Mapping Edge Properties
:PERSON :PERSON
:MARRIED
DATE=‘01-01-72’
:PERSON :PERSON
:MARRIAGE
DATE=’01-01-72’:MARRIED :MARRIED
:PERSON :PERSON
:MARRIAGE
DATE=’01-01-72’
:MARRIED_E :MARRIED_E
:MARRIED
or
- For performance reasons, the
design decision was not to have
edge properties
- Edge properties can be modeled
using auxiliary nodes

Storing Graphs in Files
Graphs can be loaded from JSG-files (local disk or HDFS)
When loading from local disk, the file must be present on all VORA nodes at the same path
JSG-format:
 Optional JSON-header with metadata # { … }
 Every line in the body is a JSON array representing one node, consisting of:
[ <node-type>, <node-id>, <properties>, <edges> ]

Creating VORA Graphs from Spark
Graph: Partitioned and un-partitioned
1. Create a BLOCK partition function:
 Recommended PARTITIONS: (number-of-VORA-nodes * cores-per-VORA-
node)
 Recommended BLOCKSIZE: 1000
2. Create a partition scheme
3. Create a graph using partition scheme and data source. Partition scheme must
get NODEID as parameter.

Create Graph

 SQL-based graph query language
 Aggregations, sorting, limits, and expressions inherited from SQL
Graph Query Language in a Nutshell
Basic Concepts
Select node type SELECT NAME FROM ACTOR USING GRAPH IMDB : Any property from one type node
Select all nodes SELECT NAME FROM ANY USING GRAPH IMDB : Any property from ANY type node
Select paths SELECT PLAYS_IN.DIRECTED_BY.NAME FROM ACTOR USING GRAPH IMDB : Traverse nodes
Node type and ID SELECT NODETYPE FROM ANY WHERE NODEID > 5 USING GRAPH IMDB : Node ID and Node Type
Built-in Graph Functions
Node degree SELECT DEGREE(A) FROM ACTOR A USING GRAPH IMDB
Distance SELECT DISTANCE(A,B) FROM ACTOR A, ACTOR B USING GRAPH IMDB WHERE…
Connected Components SELECT CONNECTED_COMPONENT(A) FROM ACTOR A USING GRAPH IMDB
Graph Pattern Matching
Check edges SELECT A.NAME, M.TITLE FROM ACTOR A, MOVIE M USING GRAPH IMDB WHERE M IN A.PLAYS_IN
Check distance SELECT A.NAME, B.NAME FROM ACTOR A, ACTOR B USING GRAPH IMDB WHERE DISTANCE(A,B) < 5

SELECT

Aggregation Functions

Graph Functions

Degree Function

Connected Component Function

Connected Component Function
SELECT A.NAME, CONNECTED_COMPONENT (STRONG A) AS
SCC, CONNECTED_COMPONENT(WEAK A) AS WCC
FROM ACTOR A USING GRAPH MOVIES
A.NAME SCC WCC
Brad Pitt 1 1
Angelina Jolie 2 1
Shah Rukh Khan 3 3

The DISTANCE function calculates the distance of the directed or undirected path between two nodes:
• Sum of weights
• Number of hops between the source and destination Node.
Distance Function (Shortest Path)
S: 1, D: 4
1-2-5-4(3)
1-5-4 (2)
1-2-3-4 (3)
1-5-2-3-4 (4)
S: A, D: B
A-B, w: 4
A-C-B, w: 3 (2+1)
A-C-D-B, w: 8 (2+3+3)
A-C-D-E-B, w: 10 (2+3+1+4)

Take 15 minutes and work through the graph notebook in Apache Zeppelin
Check It Out!

http://<DNS_NAME_OF_JUMPBOX_NODE>:9225

TOP REASONS TO CHOOSE VORA
1
Integrated Solution
Combine relational, time series, JSON, and graph
processing. No need to move data between systems
2
SQL Access
SQL access to relational, time series, JSON, and graph
computing engines
5
Metadata Persistence
Enabled by Distributed Transaction Log that can be
recovered when needed
6
Business Semantics
Out-of-the-box hierarchy processing and currency
conversion
3
Intuitive Interface
Web interface with SQL editor, data browser, and drag-and-
drop to interact with Vora computing engines
7
HANA Integration
High-performance interactive analytics across enterprise
data in HANA and Hadoop data
4
Open Consumption
Open framework supports data integration via JDBC,
OData and Restful Services

Documentation
SAP VORA website http://sap.com/vora
SAP VORA documentation https://help.sap.com/viewer/p/SAP_VORA
- Installation and Administration Guide
- Developer Guide
- Troubleshooting Guide
- Sizing Guide
Product Availability Matrix (PAM) incl. release strategy
https://apps.support.sap.com/sap/support/pam/pam.html?#pvnr=73555000100900000415
SAP Notes (component HAN-VO*) https://launchpad.support.sap.com/#/solutions/notes/
- 2303668 - SAP VORA 1.3 Release Note
- 2405200 - Release Restrictions for SAP VORA 1.3
- 2213226 - Prerequisites for installing SAP VORA: Operating Systems and Hadoop Components
- 2220859 - SAP VORA Documentation Corrections
Further help
- For product issues -> SAP customer message in components HAN-VO*
- Community-based help on Stackoverflow (no SLA, non-critical issues only) http://stackoverflow.com/questions/tagged/vora
Resources - documentation

SAP Vora CodeJam

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (19)

Ähnlich wie SAP Vora CodeJam

Ähnlich wie SAP Vora CodeJam (20)

Mehr von Vitaliy Rudnytskiy

Mehr von Vitaliy Rudnytskiy (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

SAP Vora CodeJam