BEATING THE TRAFFIC JAM 
USING NOSQL 
NoSQL Matters 
Kristoffer Dyrkorn, BEKK 
22/11/14
BEKK Public Roads Administration 
Norwegian consulting firm 
Private and public sector enterprises 
Strategy, technology, ...
Population 
(mill) 
Area 
(km2) 
Roads 
(1000 km) 
Germany 81 357 644 
Spain 46 500 683 
Norway 5 385 95 
CONTEXT
E18 Vestfold – Undrumsdal. Photo: Hans A. Rosbach
Atlanterhavsveien, www.nasjonaleturistveger.no. Photo: Harald Mowinckel
Trollstigen, www.nasjonaleturistveger.no. Photo: Steinar Skaar
Aurlandsfjellet, www.nasjonaleturistveger.no. Photo: Bjørn Andresen
ROADS ARE INFRASTRUCTURE 
BUILDING AND MAINTAINING ROADS IS EXPENSIVE 
PROPER PLANNING DEPENDS ON TRAFFIC ANALYSIS 
VEHICL...
Butunellen. Photo: Knut Opeide, Statens Vegvesen
Photo: Statens Vegvesen
Photo: Statens Vegvesen
TIME IS ESSENTIAL 
ROAD TRAFFIC IS DYNAMIC 
UPDATED TRAFFIC INFORMATION HELPS: 
• SAFETY 
• FLOW CONTROL 
• ROUTE PLANNING...
Intelligent Transport Systems: 
Measure 
& 
adjust 
road traffic
SYSTEM GOALS & REQUIREMENTS 
EASE OF INSTALLATION AND VERIFICATION OF ROADSIDE EQUIPMENT 
INCREASED DATA QUALITY 
INCREASE...
A TRAFFIC EVENT 
Voltage 
signature 
OPC-UA 
event 
Bulked 
data 
Sensor Data logger Application Storage
Sensors 
System status 
SenSsenosrosr s System 
Other 
backends 
GUI 
Vehicle info Reports 
DATA FLOW
SOLUTION ARCHITECTURE 
HTML5 GUI 
(HTTP, JSON) 
Application logic 
Support libraries 
Java VM 
OS 
Traffic 
events 
Report...
"measure_point_number": 
1601436, 
"county_id": 
16, 
"region_id": 
2, 
"server_local_timestamp": 
"2014-­‐10-­‐01T01:58:4...
HOW WE USE ELASTICSEARCH 
BULK INDEXING, JAVA API 
DATA IS INDEXED, STORED, NOT ANALYZED 
TEMPORAL SHARDING 
SPATIAL SHARD...
REPORTING 
FOR A GIVEN TIME INTERVAL, PROVIDE: 
• TOTAL VEHICLE COUNT AND AVERAGE SPEED, 
• THE 85 AND 95 PERCENTILE SPEED...
VEHICLE FLOW (10 MIN), NOV 4TH 2014
VEHICLE SPEED (1 MIN), NOV 4TH 2014
SYSTEM VALUE 
REPORTING: 
• MORE COST-EFFICIENT ROAD MAINTENANCE 
REAL TIME: 
• ROUTING OF EMERGENCY VEHICLES 
• GENERAL T...
EXPERIENCES USING ELASTICSEARCH 
ENSURE FITNESS-TO-PURPOSE 
UPGRADE CONTINUOUSLY 
REVISE THE RUN-TIME ENVIRONMENT CONTINUO...
?
THANK YOU! 
Kristoffer Dyrkorn 
BEKK 
kristoffer.dyrkorn@bekk.no
Kristoffer Dyrkorn – Beating the traffic jam using NoSQL - NoSQL matters Barcelona 2014
Kristoffer Dyrkorn – Beating the traffic jam using NoSQL - NoSQL matters Barcelona 2014
Kristoffer Dyrkorn – Beating the traffic jam using NoSQL - NoSQL matters Barcelona 2014
Kristoffer Dyrkorn – Beating the traffic jam using NoSQL - NoSQL matters Barcelona 2014
Kristoffer Dyrkorn – Beating the traffic jam using NoSQL - NoSQL matters Barcelona 2014
Nächste SlideShare
Wird geladen in …5
×

Kristoffer Dyrkorn – Beating the traffic jam using NoSQL - NoSQL matters Barcelona 2014

895 Aufrufe

Veröffentlicht am

Kristoffer Dyrkorn – Beating the traffic jam using NoSQL

Most people have experienced the boredom of being stuck in traffic. Up-to-date and credible information about congestions and detours could save us time and frustration in our everyday lives.The Norwegian Public Roads Administration is now building a new infrastructure for road traffic measurements, and the system will provide high-quality, near-realtime information as publicly available open data. The project relies heavily on NoSQL technology (Elasticsearch) for high-performance data gathering and statistical analysis.This talk will give a walkthrough of the project and the solution and show how NoSQL has helped in building an application that meets demanding requirements. Several use cases that illustrate the value of the system, both for the general public and for private companies and public institutions, will be given.

Veröffentlicht in: Daten & Analysen
  • Als Erste(r) kommentieren

Kristoffer Dyrkorn – Beating the traffic jam using NoSQL - NoSQL matters Barcelona 2014

  1. 1. BEATING THE TRAFFIC JAM USING NOSQL NoSQL Matters Kristoffer Dyrkorn, BEKK 22/11/14
  2. 2. BEKK Public Roads Administration Norwegian consulting firm Private and public sector enterprises Strategy, technology, digital services 370 employees Responsible for state and county roads Planning, construction, operation 7500 employees Spending: € 7 Billion (2013) BACKGROUND
  3. 3. Population (mill) Area (km2) Roads (1000 km) Germany 81 357 644 Spain 46 500 683 Norway 5 385 95 CONTEXT
  4. 4. E18 Vestfold – Undrumsdal. Photo: Hans A. Rosbach
  5. 5. Atlanterhavsveien, www.nasjonaleturistveger.no. Photo: Harald Mowinckel
  6. 6. Trollstigen, www.nasjonaleturistveger.no. Photo: Steinar Skaar
  7. 7. Aurlandsfjellet, www.nasjonaleturistveger.no. Photo: Bjørn Andresen
  8. 8. ROADS ARE INFRASTRUCTURE BUILDING AND MAINTAINING ROADS IS EXPENSIVE PROPER PLANNING DEPENDS ON TRAFFIC ANALYSIS VEHICLE COUNTS & WEIGHTS DECIDE PRECISE REPORTS ARE NEEDED
  9. 9. Butunellen. Photo: Knut Opeide, Statens Vegvesen
  10. 10. Photo: Statens Vegvesen
  11. 11. Photo: Statens Vegvesen
  12. 12. TIME IS ESSENTIAL ROAD TRAFFIC IS DYNAMIC UPDATED TRAFFIC INFORMATION HELPS: • SAFETY • FLOW CONTROL • ROUTE PLANNING REAL TIME DATA IS NEEDED
  13. 13. Intelligent Transport Systems: Measure & adjust road traffic
  14. 14. SYSTEM GOALS & REQUIREMENTS EASE OF INSTALLATION AND VERIFICATION OF ROADSIDE EQUIPMENT INCREASED DATA QUALITY INCREASED DATA AVAILABILITY ALL EVENTS MUST BE KEPT (NO PRE-AGGREGATION) MINIMAL LATENCY AD-HOC REPORTING SCALABILITY ROBUSTNESS
  15. 15. A TRAFFIC EVENT Voltage signature OPC-UA event Bulked data Sensor Data logger Application Storage
  16. 16. Sensors System status SenSsenosrosr s System Other backends GUI Vehicle info Reports DATA FLOW
  17. 17. SOLUTION ARCHITECTURE HTML5 GUI (HTTP, JSON) Application logic Support libraries Java VM OS Traffic events Reports (CSV, SOAP) Elasticsearch Java VM OS Request/ response Data logger N data loggers M application servers K storage servers
  18. 18. "measure_point_number": 1601436, "county_id": 16, "region_id": 2, "server_local_timestamp": "2014-­‐10-­‐01T01:58:44.330+02:00", "server_utc_timestamp": "2014-­‐09-­‐30T23:58:44.330Z", "client_utc_timestamp": "2014-­‐09-­‐30T23:58:45.229Z", "event_number": 2319762, "vehicle_type": 3, "vehicle_type_raw": "9", "vehicle_type_quality": 22228, "vehicle_number": 2319762, "speed": 80.9, "length": 16.46, "lane": 1, "gap": 10
  19. 19. HOW WE USE ELASTICSEARCH BULK INDEXING, JAVA API DATA IS INDEXED, STORED, NOT ANALYZED TEMPORAL SHARDING SPATIAL SHARDING DATA CENTER-AWARE REPLICATION NO SPECIAL OPTIMIZATIONS! RAM, CPU, DISK
  20. 20. REPORTING FOR A GIVEN TIME INTERVAL, PROVIDE: • TOTAL VEHICLE COUNT AND AVERAGE SPEED, • THE 85 AND 95 PERCENTILE SPEEDS, • IN EACH OF 5 LENGTH CATEGORIES: THE VEHICLE COUNT AND AVERAGE SPEED, • IN EACH OF 12 SPEED CATEGORIES: THE VEHICLE COUNT, ...AND ALL OF THIS FOR • EACH TRAFFIC LANE AT A MEASURE POINT, • EACH MEASURE POINT IN A REGION
  21. 21. VEHICLE FLOW (10 MIN), NOV 4TH 2014
  22. 22. VEHICLE SPEED (1 MIN), NOV 4TH 2014
  23. 23. SYSTEM VALUE REPORTING: • MORE COST-EFFICIENT ROAD MAINTENANCE REAL TIME: • ROUTING OF EMERGENCY VEHICLES • GENERAL TRAFFIC INFORMATION TO THE PUBLIC • ROUTE PLANNING ON HOLIDAYS • ROUTE PLANNING FOR PARCEL SERVICES
  24. 24. EXPERIENCES USING ELASTICSEARCH ENSURE FITNESS-TO-PURPOSE UPGRADE CONTINUOUSLY REVISE THE RUN-TIME ENVIRONMENT CONTINUOUSLY THE AGGREGATIONS MODULE IS FANTASTIC USE TOOLING (WE LIKE KOPF) MONITOR THE RESOURCE UTILIZATION THE JAVA API IS SOMETIMES COMPLEX WE ARE HAPPY!
  25. 25. ?
  26. 26. THANK YOU! Kristoffer Dyrkorn BEKK kristoffer.dyrkorn@bekk.no

×