At Scania we have around +400 000 connected vehicles, trucks and buses, roaming the Earth and continuously collecting and sending us GPS coordinates globally. To create value based on this data we need to build scalable data pipelines enabling actionable insights and applications for our company and our customers.

2. Evolution is Continuous, and so are
Big Data and Streaming Pipelines
Gustav Rånby
Data Scientist
Sarah Hantosi Albertsson
Data Scientist

3. About us
▪ The team
▪ Use cases and needs
▪ Challenges & learnings
DevOps
▪ Components, releases and pipelines
▪ Putting it together
Summing up

4. 450 000+
connected vehicles

5. Transform geospatial data into
actionable insights.

6. Can we optimize our transport flow?

7. 2020 20352022 2025 20332030
0
10
20
30
40
50
60
70
80
90
100
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
%
CO2 reduction

8. Evolution is continuous
Photo by Nikola Jovanovic on Unsplash

9. Transform continuously growing datasets, that is resilient to system failures
and transparent enough to monitor.

10. Strategies for managing credentials, input, output and checkpoint paths for
multiple environments.

11. Updated code that can continue the processing from previous versions.

12. Tests enable iterative improvements and communication around properties
of a data product.
Photo by ThisisEngineering RAEng on Unsplash

13. Use case specific post processing to deliver customizable data products.

15. Component

16. Component - name: Copy job
copy:
src: ../target/scala-2.11/{{ jar }}
dest: "{{ local_dir }}/{{ jar }}"
- name: Get Kerberos ticket
command: kinit -kt {{ kerberos_keytab }} {{ kerberos_principal }}
- name: Install jaas config worker
template:
src: jaas.conf
dest: "{{ local_dir }}/{{ jaas_config_file }}"
- name: Install jaas config driver
template:
src: jaas_driver.conf
dest: "{{ local_dir }}/{{ jaas_config_file_driver }}"
- name: Submit job
command: "{{ spark_submit_command }}"
args:
chdir: "{{ local_dir }}"
environment:
SPARK_MAJOR_VERSION: 2
register:
submit_result
ignore_errors: True
Copy the program to the cluster
Ensure authentication
Run program

17. Component

18. version 1.2 version 2.3 version 1.0 version 2.1
Release name: 1.0

19. Release
version 1.2 version 2.3 version 1.0 version 2.1
Release name: 1.0

20. Release
{
"version": "2020-05-05_16:23_1588695818",
"jobs": [{
"fms-ccp-positions-to-incremental": "1.1.1588146051_9281a60"
}, {
"positions-to-stops": "1.1.1588145988_309450b"
}, {
"stops-to-poi-stops": "1.1.1588146614_f43a667"
}, {
"dump-kafka-topic": "1.1.1588145909_771330c"
}, {
"describe-topic": "1.1.1588060837_0dd13fb"
}, {
"latest-agg-to-hive": "1.1.1587709523_7e34561"
}]
}

21. Release name: 1.0
Release name: 2.1
Release name: 3.0
Pipeline
specification

22. Pipeline specification
{"pipelines": [{
"pipeline_annotation": "2",
"release_name": “2.1",
"run_process_version": "2",
"dag": {
"start": {
"job": “positions-cleaned",
"next": ["positions-to-stops"]
},
"positions-to-stops": {
"job": "positions-to-stops",
"next": ["stops-to-poi-stops"]
},
"stops-to-locations": {
"job": "stops-to-location",
"next": [“post-process-stops", "post-process-locations"]
},
"post-process-stops": {
"job": "post-process",
"arguments": "--in stop_topic --out hdfs_path"
},
"post-process-locations": {
"job": "post-process",
"arguments": "--in location_topic --out hdfs_path"
}
}
}]
}

23. Putting it together
Code Re pos itory
Projec t
artifa ct
Int egration tes t
Upd ate Re pos itory Build – : Te s t – : Publis h
Run
COMPONENTBuild – : Te s t – : Publis h COMPONENT
RELEASE
PIPELINE
SPECIFICATION

24. Value creation is a continuum

25. Feedback
Your feedback is important to us.
Don’t forget to rate and
review the sessions.