Anzeige
Anzeige

Más contenido relacionado

Anzeige
Anzeige

Training Series - Build A Routing Web Application With OpenStreetMap, Neo4j, and Leaflet.js

  1. Build A Routing Web App With Neo4j, OpenStreetMap, & Leaflet.js William Lyon Neo4j @lyonwj lyonwj.com dev.neo4j.com/routing-workshop
  2. Building A Routing Web Application With Neo4j, OpenStreetMap, and Leaflet.js Learn how to work with geospatial data in the Neo4j graph database in this hands-on workshop. We’ll start by covering the spatial types and functions available in the Cypher query language and Neo4j then move on to spatial search operations and routing with graph algorithms. We’ll import data from OpenStreetMap and see how to use the Neo4j JavaScript driver to build a navigation web application to find routes between points of interest and addresses.
  3. 3 About Me William Lyon Developer Advocate @ Neo4j @lyonwj lyonwj.com dev.neo4j.com/routing-workshop
  4. About Me William Lyon Developer Relations Engineer, Neo4j @lyonwj lyonwj.com dev.neo4j.com/graphql-book
  5. https://twitter.com/lyonwj/status/1623731007332716544
  6. Build A Routing Web App W/ OpenStreetMap, Neo4j, & Leaflet.js ● Address / point of interest data ● Auto-complete search ● Plot shortest route between locations
  7. Build A Routing Web App With Neo4j, OSM, & Leaflet.js Agenda Modules: ● Intro to Neo4j & Spatial Cypher ● Working With OpenStreetMap Data ● Building Web Maps With Leaflet.js Hands-On exercises using: ● Neo4j AuraDB Free Tier ● Python notebook ○ Local Python environment ○ GitHub Codespaces ○ Google Colab ● JavaScript / HTML ○ Local web browser ○ GitHub Codespaces Hands-On Exercise Slides: dev.neo4j.com/routing-workshop GitHub repo: dev.neo4j.com/routing-workshop-github
  8. Build A Routing Web App With Neo4j, OSM, & Leaflet.js Pre-requisites ● Either a local Python development environment or a GitHub account to use GitHub Codespaces (recommended) or Google Colab ● Neo4j AuraDB (Free Tier) ○ Sign in and create free account: dev.neo4j.com/aura Slides: dev.neo4j.com/routing-workshop GitHub repo: dev.neo4j.com/routing-workshop-github
  9. Resources ● Slides: ○ dev.neo4j.com/routing-workshop ● Code: ○ dev.neo4j.com/routing-workshop-github
  10. 10 Intro To Neo4j & Spatial Cypher
  11. Intro to Neo4j & Spatial Cypher - Routing With Neo4j ● Slides / demo with airports ● set up Aura Free instance ● show we're going to build ● demo bloom (?)
  12. Points, Lines, Polygons, & Graphs ● Spatial Cypher functions ○ Point, distance, index ● Spatial search with Neo4j ● Routing ● QGIS + Neo4j ● Combining datasets
  13. Tobler's First Law Of Geography http://southwesterngis.blogspot.com/2014/02/toblers-first-law-of-geography.html
  14. Tobler's First Law Of Geography Minneapolis San Jose Santa Cruz San Mateo San Francisco
  15. Tobler's First Law Of Geography
  16. What Is A Graph?
  17. What Is A Property Graph?
  18. What Is A Knowledge Graph? Knowledge graphs put things in context "Things, not strings" https://blog.google/products/search/introducing-knowledge-graph-things-not/
  19. Graph Database ● Database management system (DBMS) ● Property Graph data model ● Cypher query language ● Graph analytics ● Data visualization ● Neo4j Aura database-as-a-service ● Native spatial types / index What is Neo4j? neo4j.com
  20. News As A Knowledge Graph Using The NYTimes API Requirements: ● View articles ○ Most recent ● Search articles ○ Keyword, date range ● Comments ● Save articles ● Personalized recommendations ○ My interests ○ Location-based Data from NYTimes API: developer.nytimes.com Code: github.com/johnymontana/news-graph
  21. Spectrum Of Graph Access Patterns "Local" Graph Traversals Transactional use cases Operational workloads "Global" Graph Traversals Analytics use cases Graph Data Science Graph Algorithms Graph Local Graph Global
  22. Overview of Graph Algorithms ● Path finding ○ Routing ○ A*, Breadth/Depth First Search ● Centrality ○ Most important nodes in the graph ○ PageRank, Closeness, Betweenness, Degree ● Community detection / clustering ○ Group nodes, strength of clusters ○ Louvain, Label propagation ● Node Similarity ● Machine learning ○ Link prediction ○ Node Embeddings ○ Node Classification https://neo4j.com/developer/graph-data-science/graph-algorithms/
  23. 23 Spatial Cypher Functions
  24. The Point type in Neo4j ● Support for 2D or 3D geographic (WGS84) or cartesian coordinate reference system (CRS) https://neo4j.com/docs/cypher-manual/current/syntax/spatial/ RETURN point( {latitude:49.38713, longitude:12.12711} ) ╒════════════════════════════════════════════════╕ │"point({latitude:49.38713, longitude:12.12711})"│ ╞════════════════════════════════════════════════╡ │point({srid:4326, x:12.12711, y:49.38713}) │ └────────────────────────────────────────────────┘
  25. The Point type in Neo4j ● Points can be stored as properties on nodes and relationships https://neo4j.com/docs/cypher-manual/current/syntax/spatial/ CREATE (p:PointOfInterest) SET p.name = "West Side Bakery", p.location = point( {latitude:49.38713, longitude:12.12711} ) RETURN p
  26. Spatial Functions In Cypher ● point.distance() ● point.withinBBox() ● point() https://neo4j.com/docs/cypher-manual/current/functions/spatial/ MATCH (p:PointOfInterest) WHERE point.distance( p.location, point({latitude:49.38713, longitude:12.12711}) ) < 200 RETURN p
  27. Querying The News Graph Cypher + Neo4j Browser
  28. News As A Knowledge Graph Using The NYTimes API Requirements: ● View articles ○ Most recent ● Search articles ○ Keyword, date range ● Comments ● Save articles ● Personalized recommendations ○ My interests ○ Location-based Data from NYTimes API: developer.nytimes.com Code: github.com/johnymontana/news-graph
  29. Geocoding With APOC CALL apoc.spatial.geocode('Union House, London') YIELD location { "latitude": 51.503811999999996, "description": "Union House, 182-194, Union Street, Bankside, Southwark, London Borough of Southwark, London, Greater London, England, SE1 8LB, United Kingdom", "longitude": -0.10101799277699847 } https://neo4j.com/labs/apoc/4.1/overview/apoc.spatial/apoc.spatial.geocode/
  30. Geocoding The News Graph With APOC & flat-graph GitHub Action https://github.com/johnymontana/news-graph/blob/main/.github/workflows/flat.yml
  31. Querying The News Graph Cypher + Neo4j Browser
  32. Querying The News Graph Cypher + Neo4j Browser
  33. 33 Spatial Search With Neo4j
  34. Spatial Search With Neo4j 1. Radius distance search 2. Within Bounding Box 3. Within Polygon https://github.com/johnymontana/geospatial-graph-demos
  35. OpenStreetMap Points of Interest With Daylight Earth Table https://daylightmap.org/earth/
  36. OSM Points Of Interest In Neo4j https://github.com/johnymontana/daylight-earth-graph/blob/main/POI_import.ipynb
  37. Why We Use Indexes In Neo4j ● Find a starting point for a traversal ● index free adjacency
  38. Create Point Index For Fast Search CREATE POINT INDEX poiIndex FOR (p:Point) ON (p.location) https://neo4j.com/docs/cypher-manual/current/indexes-for-search-performance/
  39. :schema
  40. NodeIndexSeek vs NodeByLabelScan PROFILE MATCH (p:Point) WHERE point.distance(p.location, point({latitude: 37.563434, longitude:-122.322255})) < 200 RETURN p Indexes help us efficiently find the starting point for a traversal 49565697 total db hits in 3649 ms 213 total db hits in 1 ms
  41. Radius Distance Search (with index!) PROFILE MATCH (p:Point) WHERE point.distance(p.location, point({latitude: 37.563434, longitude:-122.322255})) < 200 RETURN p{.*} AS point https://neo4j.com/docs/cypher-manual/current/functions/spatial/#functions-distance
  42. Cypher Bounding Box Search MATCH (p:Point) WHERE point.withinBBox( p.location, point({longitude:-122.325447, latitude: 37.55948 }), // Lower left point point({longitude:-122.314675 , latitude: 37.563596})) // Upper right point RETURN p https://neo4j.com/docs/cypher-manual/current/functions/spatial/#functions-withinBBox
  43. Cypher Bounding Box Search - Using Index PROFILE MATCH (p:Point) WHERE point.withinBBox( p.location, point({longitude:-122.325447, latitude: 37.55948 }), point({longitude:-122.314675 , latitude: 37.563596})) RETURN p https://neo4j.com/docs/cypher-manual/current/functions/spatial/#functions-withinBBox
  44. Point In Polygon Search With Index 1) Convert polygon to bounding box 2) Cypher withinBBox query 3) Filter results to within polygon https://github.com/johnymontana/geospatial-graph-demos/blob/main/src/index.html#L175-L231
  45. Working With Line Geometries Strava Data // Create Activity Nodes LOAD CSV WITH HEADERS FROM "file:///activities.csv" AS row MERGE (a:Activity {activity_id: row.`Activity ID`}) SET a.filename = row.Filename, a += row // Parse geojson geometries and create Geometry:Line nodes WITH a WHERE a.filename IS NOT NULL MERGE (n:Geometry {geom_id:a.activity_id }) MERGE (n)<-[:HAS_FEATURE]-(a) WITH n,a CALL apoc.load.json('file:///' + a.filename) YIELD value UNWIND value.features[0].geometry.coordinates AS coord // FIXME: handle multiple features! WITH n, collect(point({latitude: coord[1], longitude: coord[0]})) AS coords SET n.coordinates = coords SET n:Line
  46. Working With Line Geometries Spatial Search WITH point({latitude: $latitude, longitude: $longitude}) AS radiusCenter MATCH (g:Geometry)<-[:HAS_FEATURE]-(a:Activity) WHERE any( p IN g.coordinates WHERE point.distance(p, radiusCenter) < $radius ) RETURN *
  47. 47 Routing
  48. Overview of Graph Algorithms ● Path finding ○ Routing ○ A*, Breadth/Depth First Search ● Centrality ○ Most important nodes in the graph ○ PageRank, Closeness, Betweenness, Degree ● Community detection / clustering ○ Group nodes, strength of clusters ○ Louvain, Label propagation ● Node Similarity ● Machine learning ○ Link prediction ○ Node Embeddings ○ Node Classification https://neo4j.com/developer/graph-data-science/graph-algorithms/
  49. Airport Routing With Graph Data Science Plugin https://dev.neo4j.com/sandbox
  50. Create Projected Graph CALL gds.graph.project( 'routes-weighted', 'Airport', 'HAS_ROUTE', { relationshipProperties: 'distance' } ) YIELD graphName, nodeProjection, nodeCount, relationshipProjection, relationshipCount https://neo4j.com/docs/graph-data-science/current/management-ops/graph-catalog-ops/
  51. gds.shortestPath.dijkstra MATCH (source:Airport {iata: 'PUQ'}), (target:Airport {iata: 'HNL'}) CALL gds.shortestPath.dijkstra.stream('routes-weighted', { sourceNode: source, targetNode: target, relationshipWeightProperty: 'distance' }) YIELD index, sourceNode, targetNode, totalCost, nodeIds, costs, path RETURN nodes(path) as path https://neo4j.com/docs/graph-data-science/current/algorithms/dijkstra-source-target/
  52. Exercise ● set up neo4j aura instance
  53. Neo4j Aura Free Tier Setup Let's create a Neo4j Aura Free instance that we'll use for the rest of the workshop... Hands-On Exercise Once your Neo4j Aura instance is online you'll see the connection string (neo4j+s://xxxxx.databases.neo4j.io) Be sure to take note of the generated password! It will then take a few moments for your Neo4j Aura instance to be provisioned. Open neo4j.com/aura and sign in Select "Create a new database" button. Choose the "Free" tier. Enter a name for your Neo4j Aura instance and select "Create database" Step 1: Step 2: Step 3:
  54. 54 Working With OpenStreetMap Data
  55. 55 OpenStreetMap openstreetmap.org ● Open map data of the world ● Crowd-sourced by the community - individuals and corporate contributors ● Map tiles, geocoding, vector data, data downloads & access via API
  56. 56 Working With OpenStreetMap Data In Neo4j learn.opengeoedu.de ● Nodes - point features ○ POIs or part of more complex geometry ● Ways - line features (connect nodes) ○ roads, buildings, parks, etc ● Tags - attribute data describing a feature ○ used on nodes, ways, & relations ● Relations - organize multiple ways (or nodes) into a larger entity ○ multiple road segments that make up a bus route, multipolygon geometries https://labs.mapbox.com/mapping/osm-data-model/ https://wiki.openstreetmap.org/wiki/Elements
  57. 57 The OpenStreetMap Data Model learn.opengeoedu.de
  58. 58 Working With OpenStreetMap Data In Neo4j dev.neo4j.com/sandbox ● Example OSM dataset of Central Park in New York City ● Available in Neo4j Sandbox ● Based on neo4j-contrib/osm ● Travel guide example app: github.com/johnymontana/central- perk github.com/neo4j-graph-examples/openstreetmap
  59. Simplified OSM Routing Graph ● What if we only model intersection nodes in the graph? ● Reduces data size and simplifies our queries
  60. OpenStreetMap ● What is OpenStreetMap? ● The OSM data model ● How to get OSM data ○ overpass, QGIS plugin, neo4j osm importer, OSMNx
  61. OSMNx https://github.com/gboeing/osmnx OSMnx: Python for street networks. Retrieve, model, analyze, and visualize street networks and other spatial data from OpenStreetMap.
  62. OpenStreetMap Import Notebook Let's get our Python environment up and running and start importing OpenStreetMap data! Hands-On Exercise Step 1: ● Open notebooks/00-setup.ipynb ● Replace database credentials with your Neo4j AuraDB credentials ● Run all cells → you should see a DataFrame with node_count of 0 Step 2: ● Open notebooks/01-import.ipynb ● Replace database credentials with your Neo4j AuraDB credentials ● Run all cells up to "Exploring The Road Network In Neo4j" to import OpenStreetMap data into your Neo4j AuraDB instance
  63. Visualizing Road Networks With Neo4j Bloom ● In Neo4j AuraDB Workspace, switch to the "Explore" tab ● Load the road network into the "scene" ● Switch to coordinate layout ● Style the visualization Hands-On Exercise Let's explore our road network visually!
  64. Exercise ● add aura credentials and run notebook to import data
  65. Adding Points of Interest ● Daylight earth table ● how to import into neo4j ● connect to routing graph
  66. Adding Addresses From OpenAddresses.io https://openaddresses.io/
  67. Adding Addresses From OpenAddresses.io
  68. Adding Addresses… Hands-On Exercise ● Return to our notebook and finish running address import
  69. Routing With Graph Algorithms https://neo4j.com/docs/apoc/current/overview/apoc.algo/
  70. 70 Building Web Maps With Leaflet.js
  71. Leaflet.js https://leafletjs.com/
  72. Leaflet.js + Neo4j Let's create a Neo4j Aura Free instance that we'll use for the rest of the workshop... Hands-On Exercise ● Open web/address_routing.html ● Update with your Neo4j AuraDB credentials ● Open in web browser, you may need to run a web server: python -m http.server ● Search for routes between two addresses ● What path-finding algorithm are we using? Can we improve using the A* algorithm?
  73. Resources ● Slides: ○ dev.neo4j.com/routing-workshop ● Code: ○ dev.neo4j.com/routing-workshop-github
Anzeige