Weitere ähnliche Inhalte Ähnlich wie Processing and retrieval of geotagged unmanned aerial system telemetry (20) Mehr von DataWorks Summit/Hadoop Summit (20) Kürzlich hochgeladen (20) Processing and retrieval of geotagged unmanned aerial system telemetry 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Who am I?
Kristopher Kane – Systems Architect, Hortonworks
What is the purpose of this topic?
Commercial and military unmanned aerial systems (UAS) require that the decision
making data points be accessible remote
They produce a lot of telemetry and decisions are made based on the timeliness and
accuracy of that telemetry
Some UAS carry additional payloads for full motion video and electro optical sensors
We will cover creating an application that covers this ingesting and searching on UAS
telemetry
3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example Telemetry Types
Military grade: Key-Length-Value (KLV) – embedded telemetry as a elementary stream
PID inside an MPEG 2 transport stream - See MISB for the open specification
– interesting data points like center of frame lat/lon and video angle – thousands of keys
Hobby grade 1: MAVLINK – APM (ArduPilot Mega) telemetry that covers battery voltage,
airspeed, temperature, pitch, roll, yaw angles and acceleration, GPS data, etc
– open source
– hundreds of keys
Hobby grade 2: FrSky receiver telemetry with a subset of what you get from MAVLINK
– closed source but has been reversed engineered
– a few keys
4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Telemetry Velocity – What is the problem to be solved?
Hobbyist level small UAS
– open source hardware, software and protocol spec – ArduPilot Mega
– 150 sensor points
– 3 sets per second equals 450 points per aircraft/sec
– Yesterday, FlightAware recorded 35,662,229 location positions/hour from transponder ground
stations but this is only includes a few sensor points
Current manned aircraft
– At 0500 local today (28JUN2016), FlightAware was tracking 9,804 aircraft in the air
4,411,800 data points per second
29,412 entries per second
6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ingest Architecture
Ground Control Station
915 MHZ - Serial
Apache Knox
Undertow HTTP API
4G -TCP HTTPS
KafkaStorm
TCP HTTP
possible to embed this in Knox
custom topology entry
Other Consumers
JSON
HBase
spatial missions
Solr
telemetry
Aircraft
7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ingest Architecture - Spatial based load distribution
* Alerts like congested airspace will happen here
Partition based on geohash prefix as a simple dimension for distribution
Increase prefix characters as load increases. This works for:
– Kafka broker partitions
– Storm bolts - a couple grouping options here
– Pre-split HBase regions
8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Geohash
Created for a URL shortening service in 2008
Encodes latitude and longitude into a string – two dimensions into one which is
important for HBase key scanning
My house is at decimal degrees: 35.45380534178051,-78.8184043765068
geohash: dnrcwqjk
More characters = more precision
Geohash doesn’t follow the base-32 RFC - read about that and don’t lose sleep like I did
9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Geohash
http://geohash.gofreerange.com/
10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Geohash Decoding
My house (area) geohash: dnrc
Using base-32:
d (base-32) = 12 (base-10) = 01100 (binary)
n = 20 = 10100
r = 23 = 10111
c = 11 = 01011
Binary representation = 01100-10100-10111-01011
From the geohash Wikipedia article
11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Geohash Decoding – House Latitude
01100-10100-10111-01011
Bits alternate longitude (even) and latitude (odd) respectively, left to right.
Latitude = 1011001001 bit min mid max latitude
1 -90.000 0.000 90.000 45.000
0 0.000 45.000 90.000 22.500
1 0.000 22.500 45.000 33.750
1 22.500 33.750 45.000 39.375
0 33.750 39.375 45.000 36.562
0 33.750 36.562 39.375 35.156
1 33.750 35.156 36.562 35.859
0 35.156 35.859 36.562 35.508
0 35.156 35.508 35.859 35.332
1 35.156 35.332 35.508 35.420
12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why Apache Knox?
Let Knox handle authentication - It will provide the ‘user.name’ query parameter appended to your
parameter set
KNOX-670 allows for the hosting of simple web applications
Custom APIs can be added to your topology with only a few lines of XML for simple cases:
<rules>
<rule dir="IN" name="DRONE-TELEMETRY/drone-telemetry/inbound" pattern="*://*:*/**/drone-telemetry/{path=**}?{**}">
<rewrite template="{$serviceUrl[DRONE-TELEMETRY]}/{path=**}?{**}"/>
</rule>
</rules>
<service role="DRONE-TELEMETRY" name="DRONE-TELEMETRY" version="0.0.1">
<routes>
<route path="/drone-telemetry/**"/>
</routes>
</service>
data/services/drone-telemetry/0.0.1/service.xml
data/services/drone-telemetry/0.0.1/rewrite.xml
14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Egress Architecture - User Query Access
Apache Knox
Undertow
HTTP API
TCP HTTP
User – mobile app or
browser based
HTTPS – Twitter Oauth/LDAP
HBase
spatial missions
Solr
telemetry
15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Well Known Text Format
Both the Solr and HBase examples following use the Well Known Text format to specify
polygons
This standard requires the points to be in counter clockwise order
Topologically closed - make the last point the first point
16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HBase Queries – General Idea
Encode two dimensions (latitude and longitude) into one (geohash)
Use geohashes as row scanning filters
Use spatial libraries against the latitude and longitude column families for accurate
calculations like distance and points inside geometries
Rely on external libraries like JTS or Spatial4j
You have to implement everything yourself
17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HBase Queries
The Undertow API will accept a bounding box set of lat/lon coordinates from our user’s
map
The web application will format that in WKT standard for the follow on geometry classes
Ensure you close the polygon with the start point
POLYGON ((-78.818951 35.4535516, -78.818960 35.4535516, -
78.818951 35.4535560, -78.818941 35.4535512, -78.818951
35.4535516))
Geometry query = reader.read(polygon);
WKT
18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HBase Query - Show Me Missions In This Polygon (map)
Get the polygon centroid and compute minimum boundaries by reversing the geohash
into latitude and longitude
Scan each geohash as a PrefixFilter to get candidate rows
*This code is from Chapter 8 of HBase In Action – I highly
recommend this book.
GeoHash[] minimumBoundingPrefixes(Geometry query)
...
public Set<QueryMatch> query(Geometry query) throws IOException {
GeoHash[] prefixes = minimumBoundingPrefixes(query);
et<QueryMatch> ret = new HashSet<QueryMatch>();
for (GeoHash prefix : prefixes) {
byte[] p = prefix.toBase32().getBytes();
Scan scan = new Scan(p);
Filter filters = new FilterList(new PrefixFilter(p), new PageFilter(500L));
19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HBase Query - Show Me Missions In This Polygon (map)
Check the returned candidate rows by doing a point lookup of the lat/long CFs against
our map provided polygon
*This code is from Chapter 8 of HBase In Action – I highly
recommend this book.
for (Iterator<QueryMatch> iter = ret.iterator(); iter.hasNext(); ) {
QueryMatch candidate = iter.next();
Coordinate coord = new Coordinate(candidate.lon, candidate.lat);
Geometry point = factory.createPoint(coord);
if (!query.contains(point)) {
iter.remove();
exclusionCount++;
}
}
‘query’ is our polygon from the map.
20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Solr Query - Show Me Missions In This Polygon (map)
&fq={!field f=geo_location}
Intersects(POLYGON(
(-80.12878 35.85121,
-80.12878 34.50429,
-77.93152 34.50429,
-77.93152 35.85121,
-80.12878 35.85121)))
=*:*&sort=gcs_time%20desc
35.85121, -80.12878
34.50429, -80.12878 34.50429, -77.93152
35.85121, -77.93152
21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Solr Query - Show Me Missions In This Polygon (map)
"response":
….
[ { "mission_name":"kris-02",
"gcs_time":1466388208,
"latitude":35.4534768,
"roll":-0.00834990758449,
"user.name":"tom",
"ingest_date":"2016-06-20T02:03:28Z",
"yaw":0.187950670719,
"pitch":-0.0191747546196,
"longitude":-78.8189403},…]}
35.85121, -80.12878
34.50429, -80.12878 34.50429, -77.93152
35.85121, -77.93152
22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Solr Query – Point in Polygon breakdown
&select?fl=mission_name,latitude,longitude,gcs_time
&fq={!field f=geo_location}Intersects(POLYGON((-80.12878 35.85121, -80.12878 34.50429, -
77.93152 34.50429, -77.93152 35.85121, -80.12878 35.85121
&q=*:*
&sort=gcs_time desc
<fieldType name="location_rpt"
class="solr.SpatialRecursivePrefixTreeFieldType"
spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
geo="true"
distErrPct="0.025"
maxDistErr="0.001"
distanceUnits="kilometers"/>
Polygons require this.
‘geo_location’ is of this type.
Don’t forget to add JTS libs to Solr server!
23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HBase Query – Show me current missions
The HBase table is created with a TTL and the row key is the mission name
This table expects high velocity but will only store one row per mission
Data is ‘UPSERT’
HBase has a cool new feature that allows for no compaction TTL clean up which is
perfect for this table’s purpose: https://issues.apache.org/jira/browse/HBASE-14468
Simply row scan the table for the mission list
24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Solr Query – Show me current missions
Provides a count list of the facet fields
select?facet.field=mission_name
&facet=on
&fq=ingest_date:[NOW-1MINUTE TO NOW]
&q=*:*
&rows=0
&start=0
http://localhost:5000/solr/mavlink/select?facet.field=mission_name&facet=on&&fq=ingest_date:[NOW-1MINUTE%20TO%20NOW]&indent=on&q=*:*&rows=0&start=0&wt=json
25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Solr Query – Distance and temporal filtering with sorting
fq=ingest_date:
[NOW-10SECOND TO NOW]
AND
{!geofilt pt=35.4,-78.81 sfield=latlon d=20}
&q=*:*
&sort=gcs_time desc
Show me:
Mission data points in the last 10 seconds,
within 20 km of my current position
and sorted by most recent ground station time
Want a unique list of missions?
facet.field=mission_name
&facet=on
&rows=0
http://localhost:5000/solr/mavlink/select?fl=mission_name,ingest_date,latitude,longitude,user.name,pitch,yaw,roll,gcs_time&fq=ingest_date:[NOW-
10SECOND%20TO%20NOW]AND{!geofilt%20pt=37.3,-121.92%20sfield=latlon%20d=20}&indent=on&q=*:*&sort=gcs_time%20desc&wt=json