For data, and data science, to be the fuel of the 21th century, data driven applications should not be confined to dashboards and static analyses. Instead they should be the driver of the organizations that own or generates the data. Most of these applications are web-based and require real-time access to the data. However, many Big Data analyses and tools are inherently batch-driven and not well suited for real-time and performance-critical connections with applications. Trade-offs become often inevitable, especially when mixing multiple tools and data sources. In this talk we will describe our journey to build a data driven application at a large Dutch financial institution. We will dive into the issues we faced, why we chose Python and pandas and what that meant for real-time data analysis (and agile development). Important points in the talk will be, among others, the handling of geographical data, the access to hundreds of millions of records as well as the real time analysis of millions of data points.
5. Real-time, data driven app?
•No store and retrieve;
•Store, {transform, enrich, analyse} and retrieve;
•Real-time: retrieve is not a batch process;
•App: something your mother could use:
SELECT attendees !
FROM!pydataberlin2014 !
WHERE password = '1234';
33. Who has my data?
•First iteration was a (pre)-POC, less data (3GB vs
500GB);
•Time constraints;
•Oeps:
34. Who has my data?
•First iteration was a (pre)-POC, less data (3GB vs
500GB);
•Time constraints;
•Oeps:
import pandas as pd!
...!
source_data = pd.read_csv("data.csv", …)!
...!
def get_data(postcodes, dates):!
result = filter_data(source_data, postcodes, dates)!
return result
35. Advantage of “everything is a df”
Pro:
•Fast!!
•Use what you know
•NO DBA’s!
•We all love CSV’s!
!
!
!
36. Advantage of “everything is a df”
Pro:
•Fast!!
•Use what you know
•NO DBA’s!
•We all love CSV’s!
!
!
!
Contra:
•Doesn’t scale;
•Huge startup time;
•NO DBA’s!
•We all hate CSV’s!
37. If you want to go down this path
•Set the dataframe index wisely;
•Align the data to the index:
!
•Beware of modifications of the original dataframe!
source_data.sort_index(inplace=True)
46. Issues?!
•With a radius of 10km, in Amsterdam, you get 10k
postcodes.You need to do this in your SQL:
!
!
!
•Index on date and postcode, but single queries
running more than 20 minutes.
SELECT * FROM datapoints !
WHERE !
date IN date_array !
! ! AND !
! ! ! postcode IN postcode_array;
47. Postgres + Postgis (2.x)
PostGIS is a spatial database extender for PostgreSQL.
Supports geographic objects allowing location queries
SELECT *!
FROM datapoints!
WHERE ST_DWithin(lon, lat, 1500)!
AND dates IN ('2013-02-30', '2013-02-31');!
-- every point within 1.5km !
-- from (lat, lon) on imaginary dates
49. Steps to solve it
1. Align data on disk by date;
2. Use the temporary table trick:
!
!
!
!
3. Lose precision: 1234AB→1234
4. (Compression)
CREATE TEMPORARY TABLE tmp (postcodes STRING NOT NULL PRIMARY KEY);!
INSERT INTO tmp (postcodes) VALUES postcode_array;!
!
SELECT * FROM tmp!
JOIN datapoints d!
ON d.postcode = tmp.postcodes!
WHERE!
d.dt IN dates_array;
50. GoDataDriven
We’re hiring / Questions? / Thank you!
@gglanzani
giovannilanzani@godatadriven.com
Giovanni Lanzani
Data Whisperer