This document discusses Apache Arrow, a new open source project that aims to standardize in-memory columnar data representations. It will enable faster data sharing and analysis across systems by avoiding costly serialization. The document outlines how Arrow focuses on CPU efficiency through cache locality, vectorized operations, and minimal overhead. It provides examples of how Arrow could improve I/O performance for Python tools interacting with big data systems and the Feather file format developed using Arrow. Language bindings for Arrow are under development for Python, R, Java and other languages.

1. 1
©
Cloudera,
Inc.
All
rights
reserved.
Next-­‐genera;on
Python
Big
Data
Tools,
powered
by
Apache
Arrow
Wes
McKinney
@wesmckinn
SF
Big
Analy;cs
Meetup,
2016-­‐04-­‐05

2. 2
©
Cloudera,
Inc.
All
rights
reserved.
Me
• Data
Science
Tools
at
Cloudera,
formerly
DataPad
CEO/founder
• Serial
creator
of
structured
data
tools
/
user
interfaces
• Wrote
bestseller
Python
for
Data
Analysis
2012
• Open
source
projects
• Python
{pandas,
Ibis,
statsmodels}
• Apache
{Arrow,
Parquet,
Kudu
(incuba;ng)}
• Mostly
work
in
Python
and
Cython/C/C++

3. 3
©
Cloudera,
Inc.
All
rights
reserved.
In
process:
Python
for
Data
Analysis:
2nd
Edi4on
Coming
late
2016
/
early
2017

4. 4
©
Cloudera,
Inc.
All
rights
reserved.
Python
+
Big
Data:
The
State
of
things
• See
“Python
and
Apache
Hadoop:
A
State
of
the
Union”
from
February
17
• Areas
where
much
more
work
needed
• Binary
file
format
read/write
support
(e.g.
Parquet
files)
• File
system
libraries
(HDFS,
S3,
etc.)
• Client
drivers
(Spark,
Hive,
Impala,
Kudu)
• Compute
system
integra;on
(Spark,
Impala,
etc.)

5. 5
©
Cloudera,
Inc.
All
rights
reserved.
Apache
Arrow
Many
slides
here
from
my
joint
talk
with
Jacques
Nadeau,
VP
Apache
Arrow

6. 6
©
Cloudera,
Inc.
All
rights
reserved.
Arrow
in
a
Slide
• New
Top-­‐level
Apache
Sofware
Founda;on
project
• Announced
Feb
17,
2016
• Focused
on
Columnar
In-­‐Memory
Analy;cs
1. 10-­‐100x
speedup
on
many
workloads
2. Common
data
layer
enables
companies
to
choose
best
of
breed
systems
3. Designed
to
work
with
any
programming
language
4. Support
for
both
rela;onal
and
complex
data
as-­‐is
• Developers
from
13+
major
open
source
projects
involved
• A
significant
%
of
the
world’s
data
will
be
processed
through
Arrow!
Calcite
Cassandra
Deeplearning4j
Drill
Hadoop
HBase
Ibis
Impala
Kudu
Pandas
Parquet
Phoenix
Spark
Storm
R

7. 7
©
Cloudera,
Inc.
All
rights
reserved.
Apache
Arrow:
What
is
it?
• hkp://arrow.apache.org
• Not
a
piece
of
sofware,
exactly!
• A
standardized
in-­‐memory
representa;on
for
columnar
data
• Enables
• Suitable
for
implemen;ng
high-­‐performance
analy;cs
in-­‐memory
(think
like
“pandas
internals”)
• Cheap
data
interchange
amongst
systems,
likle
or
no
serializa;on
• Flexible
support
for
complex
JSON-­‐like
data
• Targets:
Impala,
Kudu,
Parquet,
Spark

8. 8
©
Cloudera,
Inc.
All
rights
reserved.
Focus
on
CPU
Efficiency
1331246660
1331246351
1331244570
1331261196
3/8/2012 2:44PM
3/8/2012 2:38PM
3/8/2012 2:09PM
3/8/2012 6:46PM
99.155.155.225
65.87.165.114
71.10.106.181
76.102.156.138
Row 1
Row 2
Row 3
Row 4
1331246660
1331246351
1331244570
1331261196
3/8/2012 2:44PM
3/8/2012 2:38PM
3/8/2012 2:09PM
3/8/2012 6:46PM
99.155.155.225
65.87.165.114
71.10.106.181
76.102.156.138
session_id
timestamp
source_ip
Traditional
Memory Buffer
Arrow
Memory Buffer
• Cache
Locality
• Super-­‐scalar
&
vectorized
opera;on
• Minimal
Structure
Overhead
• Constant
value
access
• With
minimal
structure
overhead
• Operate
directly
on
columnar
compressed
data

9. 9
©
Cloudera,
Inc.
All
rights
reserved.
High
Performance
Sharing
&
Interchange
Today With Arrow
• Each system has its own internal
memory format
• 70-80% CPU wasted on serialization
and deserialization
• Similar functionality implemented in
multiple projects
• All systems utilize the same memory
format
• No overhead for cross-system
communication
• Projects can share functionality (eg,
Parquet-to-Arrow reader)
Pandas Drill
Impala
HBase
KuduCassandra
Parquet
Spark
Arrow Memory
Pandas Drill
Impala
HBase
KuduCassandra
Parquet
Spark
Copy & Convert
Copy & Convert
Copy & Convert
Copy & Convert
Copy & Convert

10. 10
©
Cloudera,
Inc.
All
rights
reserved.
Big
Data
Systems:
Poor
Python
IO
performance
h9p://wesmckinney.com/blog/pandas-­‐and-­‐apache-­‐arrow/

11. 11
©
Cloudera,
Inc.
All
rights
reserved.
Real
World
Example:
Feather
File
Format
for
Python
and
R
• Problem:
fast,
language-­‐
agnos;c
binary
data
frame
file
format
• Wriken
by
Wes
McKinney
(Python)
Hadley
Wickham
(R)
• Read
speeds
close
to
disk
IO
performance
Arrow array 0
Arrow array 1
…
Arrow array n
Feather
metadata
Feather file
Apache Arrow
memory
Google
flatbuffers

12. 12
©
Cloudera,
Inc.
All
rights
reserved.
Real
World
Example:
Feather
File
Format
for
Python
and
R
library(feather)
path
<-­‐
"my_data.feather"
write_feather(df,
path)
df
<-­‐
read_feather(path)
import
feather
path
=
'my_data.feather'
feather.write_dataframe(df,
path)
df
=
feather.read_dataframe(path)
R
Python

13. 13
©
Cloudera,
Inc.
All
rights
reserved.
Apache
Parquet:
Binary
columnar
storage
format
• I
just
became
a
Parquet
commiker!
• github.com/apache/parquet-­‐cpp
• Python
users
will
soon
be
able
to
read
Parquet
files
via
PyArrow
• parquet-­‐cpp
<-­‐>
PyArrow
<-­‐>
pandas

14. 14
©
Cloudera,
Inc.
All
rights
reserved.
Language
Bindings
• Target
Languages
• Java
(beta)
• CPP
(underway)
• Python
&
Pandas
(underway)
• R
• Julia
• Ini;al
Focus
• Read
a
structure
• Write
a
structure
• Manage
Memory

15. 15
©
Cloudera,
Inc.
All
rights
reserved.
pandas
and
Arrow
in
context

16. 16
©
Cloudera,
Inc.
All
rights
reserved.
RPC
&
IPC:
Moving
Data
Between
Systems
RPC
• Avoid
Serializa;on
&
Deserializa;on
• Layer
TBD:
Focused
on
suppor;ng
vectored
io
• Scaker/gather
reads/writes
against
socket
IPC
• Alpha
implementa;on
using
memory
mapped
files
• Moving
data
between
Python
and
Drill
• Working
on
shared
alloca;on
approach
• Shared
reference
coun;ng
and
well-­‐defined
ownership
seman;cs

17. 17
©
Cloudera,
Inc.
All
rights
reserved.
Execu;ng
data
science
languages
in
the
compute
layer
UI
Ibis, SQL, Spark API, …
Compute
Analytic SQL, Spark, MapReduce
Storage
HDFS, Kudu, HBase
Python,
R, Julia, …?

18. 18
©
Cloudera,
Inc.
All
rights
reserved.
Real
World
Example:
Python
With
Spark,
Drill,
Impala
in partition 0
…
in partition
n - 1
SQL Engine
Python
function
input
Python
function
input
User-supplied
Python code
output
output
out partition 0
…
out partition
n - 1
SQL Engine

19. 19
©
Cloudera,
Inc.
All
rights
reserved.
What’s
Next
• Parquet
for
Python
&
C++
• Using
Arrow
as
intermediary
• Available
IPC
Implementa;on
• Spark,
Drill
Integra;on
• Faster
UDFs,
Storage
interfaces

20. 20
©
Cloudera,
Inc.
All
rights
reserved.
Apache
Arrow
in
prac;ce

21. 21
©
Cloudera,
Inc.
All
rights
reserved.
Get
Involved
• Join
the
community
• dev@arrow.apache.org
• Slack:
hkps://apachearrowslackin.herokuapp.com/
• hkp://arrow.apache.org
• @ApacheArrow

22. 22
©
Cloudera,
Inc.
All
rights
reserved.
Thank
you
Wes
McKinney
@wesmckinn
Views
are
my
own