Pipedrive uses AWS services like S3, Redshift, RDS, EC2, and Kinesis to support its data warehouse on AWS. The data warehouse processes over 100,000 backups daily totaling over 3-4TB of data to provide metrics, analytics, and business intelligence to executives and product teams. Disparate data sources are processed into a unified 3TB data set using Hadoop and Spark stacks for ETL. Security measures include VPC isolation, IAM roles, encryption at rest and in transit using KMS and S3 encryption.

1. Pipedrive DW on AWS
November 3, 2016
Erkki Suurna

2. Talking points
DW role in Pipedrive
AWS services PDW use
DW infrastructure
VPC, S3, Redshift, RDS, EC2, Kinesis, ELB
Security - KMS, IAM, S3 encryption
Hadoop and Spark stack
Self developed ETL in Python (300+ tasks running daily)

3. PDW main goal
Support data driven organisation
SaaS KPI metrics reporting to executive level
Product instrumentation and analysis
Feedback to end user
Business intelligence - data acquisition and transformation into meaningful
information for business analysis

4. PDW data platform
Disparate data sources are processed into one unified trusted data level (3TB)
25+ models and aggregates
100+ tables
Sensitive data is obfuscated
Encourage to use aggregated models instead of many canned reports
Process daily 100+k backups 330GB compressed => 3-4 TB data

5. AWS services PDW use - VPC
VPC
Logically isolated section, where you can define your virtual network
Every PDW service is in separate subnet and has separate security groups
US east region, 5 availability zones
Everything is closed by default
Infra is not permanent, script everything, because you will recreate it one day

6. AWS services PDW use - S3
Simple Secure Storage
Central point to all services
Not file system, more like endless storage
Encryption - AES-256
Lifecycle policy and storage classes
Distribute filenames in bucket by filename prefix for better performance (100TPS)
Multipart upload.

7. AWS services PDW use - Redshift and RDS
Redshift 3 node 6TB cluster
Based on Postgres
MPP - great product for analytical queries
Automatic backup
Scale in compute dense mode up to 300 TB, in storage dense mode up to 2 PB
No stored procedures nor functions
Load data from S3
Relational Database Service
Multi AZ, automatic backup and failover
ETL and Spark backend use Postgres as metastore
Snapshots of Pipedrive app backend Mysql-s

8. AWS services PDW use - EC2
Elastic Compute Cloud
Script all instances
Pre-built images (we use mainly Amazon Linux because it is managed by Amazon)
Spot instances (instance vs vCPU)
Auto Scaling Group
Access Control
KMS to encrypt and decrypt secrets
Identity and Access Management (IAM)
Security group as virtual firewall on instance level

9. AWS Kinesis
Scalable buffer
A shard is base throughput unit
1MB/sec data input
2MB/sec data output
up to 1000 PUT records/sec
24h retention

10. Hadoop and Spark stack
HDFS - new DW concept
Namenode is micro instance
2x datanode provide 10+TB storage
Spark - lightning fast processing cluster
ETL cluster in Yarn mode
Ad-hoc cluster in standalone mode
APIs
Python, Scala, Java, SQL, GraphX, Streaming, MLib
Formats
Source data in JSON
Destination data in parquet
Change deltas in CSV for Redshift import
Current setup
r3.8xlarge ( 32Core 244GB memory 2X320 GB ssd,
$2.66, spot saves 80-90%)

11. Data visualisation tools
Tableau
Desktop licence for development
Tableau Online serve interactive reports to end users
Re:dash
Fast and simple data visualisation
Grafana
Technical dash for infra monitoring
Zeppelin
Part or Spark stack
25+ interpreters available, currently we use Markdown, Shell, Scala, Python, SQL

12. Spark Zeppelin
demo
Interpreters
Shell
SQL
Python
Scala

13. Next steps
Spark
Spark Streaming POC
Kafka POC
Enable real-time dashboard on top of ElasticSearch + Kibana
POC based on graphic intensive machines (utilize GPU in Spark)
Alluxio - data storage in memory

14. Q & A