Presented at the AWS Community Day in the Bay Area on September 13, 2019 at the Computer History Museum.

1. Cruising in Data Lake:
From zero to scaleHERE Technologies | 2019
Sneha Chaphalkar, Nikita Voronin

2. HERE in numbers
2
Countries mapped
200
30+
Years of
experience
transforming
location
technology
8,000+Employees in 56 countries
focused on delivering the world’s best
map and location technologies
HERE Maps on board of
100Mvehicles and counting
28TB map data
collected
perday
4of5
In-car
navigation
systems in Europe
and North America
use HERE maps
+ collecting
data for
maps
400 HERE cars
3D data points
per second per car
700,000

3. “Origins” for our Data Lake – HERE HD Live Map
© 2019 HERE3
 Critical role in the Self-driving cars
ecosystem.
 Four key building blocks to enable
autonomous vehicles, sensing (the eyes of
the vehicle), perception, decision making
and high definition maps, what we call the
brain of the vehicle.
 “Map as an extended sensor” High
definition maps provide the vehicle
more strategic insight, allowing the car
to make more proactive decisions.

4. “Origins” for our Data Lake
 Multi-Layer product with data enriched from multiple sources with
petabyte scale data
 Complex graph of batch/streaming jobs making it difficult to
untangle and explore data dependencies.
Several valuable feeds:
 Sources feed;
 Predictive/Training model feeds
 Automated process feeds;
 Human-in-loop interactions.
© 2019 HERE4

5. Lake Range
© 2019 HERE5
Processing
data
25+Sub-systems
30+TB Active
data
Geo-spatial
Regions
Multi-Regions
Slow Data Cadence
every 1-2 hour
daily
Fast Data Cadence
Under 1 min
Process
data
90+TB Total
history
Processdata
Data-As-A-Service
Process Metrics
Business Metrics
Live Content Metrics
Quality Metrics

6. What is Data lake?
 Centralized Repository to store all relational and non-
relational data;
 No rigid design: raw data with schema on-read capability;
 Multi-faceted use-cases:
 Exploratory analysis;
 Predictive analytics;
 Machine Learning;
 Batch reporting;
 Reverse lookup.
© 2018 HERE | HERE Internal Use Only6
 Centralized Repository to store all relational and non-
relational data;
 No rigid design: Bring raw data and also prepare data for
analysis
 Multi-faceted use-cases:
 Batch reporting;
 Exploratory analysis;
 Predictive analytics;
 Machine Learning;
 Reverse lookup.

7. Foundation Blocks for Data analytics platform
© 2019 HERE7
Transformers Storage
Query Engines/
Aggregators

8. Mile Marker 0
Challenges for Data Warehouse
 Blending across multiple DW’s
 ETL jobs with fixed schema less
flexibility
 Long cycle time for changes in
columns due to multi—
teams/pipelines
 Single point of failure
 DB maintenance and tuning
 Contention for reads/writes
 High costs
© 2018 HERE8
Data Providers
BI Queries
High Cost
DW

9. Mile Marker 50
© 2019 HERE9
AWS EMR Spark Transformers &
AWS S3 for storage
 Easy to produce with Spark SQL
 No Schema maintenance (“just put
more columns”)
 “Readers” are separated from “Writers”
• Easy to scale
• No need to orchestrate
• No competition for resources*
 Easy to share
 Good compression ratio
BI Queries
Storage
Query Engies/
Aggregators
Transformers
Query Engine/
Aggregators

10. Buoys
Uniform data
Single format + single interface
Frequent incremental updates
Data comes in different forms and formats, from different sources. Update daily.
One Table = One Source
Preserve the external references to combine the data later on.
Consistency & Availability first
Data is useless if you can’t use it.
Security & compliance
Compliance with corporate standards + Hello, GDPR!
© 2019 HERE10
Guidelines for Storage in AWS S3 Data lake
Transformers Storage
Query Engines/
Aggregators

11. Log or History
• Records every change
• Append writes
• Partitioned by date*
© 2019 HERE
* -- or any other monotonically increasing key
11
Storage
Latest state
• No duplicates
• Partitioned by hash
• Copy-on-write
Lookups, aggregates
• Materialized view
• Partitioned by date*
Principles
Versioned change sets
Immutable data
Dependency tracking
“Additional content v5 uses base
content v8”
Compatibility
If two sources have matching
dependencies, then they are
“compatible”.
Append!
Merge!
Aggregate!
Query Engines/
Aggregators
Mile Marker 50 Transformers Storage

12. table/
date=2019-05-17/
01:15.parquet.snappy
01:30.parquet.snappy
…
12:45-parquet.snappy
date=2019-05-16/ daily.parquet.snappy
• Know your customers
• How fresh should be really?
• How much are they willing to pay?
• Is it worth it?
• Fast avenue
• Writes frequently → many files → slow reads;
• Limited scope. Example: past 24h only;
• Cadence: every 15 minutes.
• Slow avenue
• Buffers → few files → faster reads;
• Full scope
• Cadence: every 4h.
• Combine both!
• Append often & compress daily.
© 2019 HERE12
Fast & Slow data pipelines
Speed Boosters StorageTransformers
Query Engines/
Aggregators

13. In a nut-shell
© 2019 HERE13
Storage
Query Engines/
Aggregators
Low costHigh Cost
Transformers

14. What’s next ?
Auto-pilot
© 2019 HERE14

15. Cruising Auto-Pilot
Libraries to integrate to Data Lake
 Data compression
 Data Partitioning scheme
 Data Dependencies tracking
Monitoring and Alerting
 Centralized solutions : ELK, Splunk
 Latencies monitoring
© 2019 HERE15
Bring Your Own Data – Create a platform
Framework for boilerplate abstractions
Built-in data-design patterns
 Historical tracking
 Latest and Greatest state of data
Cost optimization
 Auto-scaling techniques
 Query cost
 TTL on storage

16. Questions?
© 2019 HERE16