Space Debris are defunct objects in space, including old space vehicles or fragments from collisions. Space debris can cause great damage to functional space ships and satellites. Thus detection of space debris and prediction of their orbital paths are essential. The talk shows a Python based infrastructure for storing space debris data from sensors and high-throughput processing of that data.
PyData Seattle (26. Juli 2015)
http://seattle.pydata.org/schedule/presentation/35/
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
High Throughput Processing of Space Debris Data
1. High-Throughput Processing of
Space Debris Data
Andreas Schreiber, Michael Meinel
German Aerospace Center (DLR)
Simulation and Software Technology,
Berlin / Cologne, Germany
PyData Seattle
July 26, 2015
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 1
2. • Space Debris
• BACARDI
• Skynet
Outline
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 2
3. Major research areas of DLR
• Aeronautics
• Space
• Transportation
• Energy
• Security
Software research and dev.
• Software engineering, HPC,
distributed systems, …
DLR
Aerospace Research Center, Space Agency
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 3
4. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 4
Space Debris
5. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 5
Lower Orbit
300 – 2000 km
Geostationary Orbit
≈36,000 km
6. Space Flight
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 6
7. • Almost 5,000 rockets launched
• More than 6,000 satellites placed in orbit
• About 1,000 active satellites today
(that’s 7% of known objects in orbit)
About 50 Years of Space Flight
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 7
8. Defunct objects in space
• Disused satellites & rocket stages
• Fragments from explosions &
collisions
• Released object (tools etc.)
• Slag of solid-fueled rockets
• … and other sources of small
particles
Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 8
9. Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 9
January 2007:
FengYun-1C
February 2009:
Collision
Iridium-33 &
Kosmos-2251
10. Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 10
Iridium-33 & Kosmos-2251:
Debris fields after 50 minutes
Fengyun-1C debris one month
after its disintegration
11. 16,300 Objects in Catalogue
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 11
12. 29,000 Objects Larger than 10 cm
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 12
13. 750,000 Objects Larger than 1 cm
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 13
14. 150M Objects Larger than 1 mm
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 14
15. Impact
Laboratory Impact Aluminum Sphere / Block
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 15
16. Impact
Real Debris Object on a Satellite Solar Panel
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 16
17. Distribution of Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 17
Source: NASA Orbital Debris Quarterly News 2/2012
ISS
Many Earth Observation
Satellites (e.g., Envisat)
TerraSAR-X
18. Collision Prediction
International Space Station
DLR.de • Chart 18 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015
19. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 19
20. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 20
http://rbth.com/news/2015/07/26/iss_successfully_ducks_from_space_debris_48023.html
21. Detecting Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 21
Sources: FhG FHR, AIUB, ESA
Optical Radar Laser
22. SMARTnet
• Theory: complete coverage
with 3 locations
• Northern / southern hemisphere
for compensating seasonal
variations (6 locations)
• Telerobotical operation
• Optimized scheduler for all
telescopes
Operational Collision Avoidance
Global network for monitoring the geostationary ring
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 22
23. Operational Collision Avoidance
Global network for monitoring the geostationary ring
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 23
24. Object Identification with Optical Measurements
DLR.de • Chart 24 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015
25. • Separation of real / false tracklets above threshold of loss
function (chi-squared distribution)
• Filter rate depends on accuracy, time difference, survey
strategy, …
Object Correlation
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 25
26. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 26
BACARDI
27. Objective
• Database with preferably high completeness and high
accuracy
• Primary source: sensor data and operator data
• Secondary source: externally generated ephemerides
BACARDI
Backbone Catalogue of Relational Debris Information
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 27
28. Science and Research
• Database of more than 1M objects
• High-performance computational…
• object correlation
• orbit determination
• propagation
• object identification
• detection of maneuvers and
fragmentations
BACARDI
Backbone Catalogue of Relational Debris Information
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 28
29. Mission Support
• Orbit information
• Collision prediction
• Re-entry prediction
Real-time operation
BACARDI
Backbone Catalogue of Relational Debris Information
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 29
30. BACARDI Overview
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 30
31. BACARDI Overview
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 31
Sensor network
BACARDI
Sensor
scheduler
User
Groups
External orbital /
object information
Optical Radar Laser
Database
User Interface
Processes
32. BACARDI
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 32
33. External Data Sources
Example: space-track.org
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 33
34. BACARDI
System Components
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 34
RDBMS
Cache
Import
Processing
Python
FORTRAN
Middleware
Security
Traceability
• Simple
• Parallel
• Fast
• Huge number
Export
35. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 35
Skynet
36. Specialized middleware with following objectives
• Decentralized message queues
• Scalable, self-organizing network
• Minimal network overhead
• Platform independent
• Operation: Linux
• Development: Windows, Mac OSX
• Automatic provenance recording
Skynet
Network for surveillance of the sky
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 36
37. ZeroMQ
• Decentralized network infrastructure and messaging
Protocol Buffers
• Highly efficient serialization of data
SQLAlchemy
• Database mapping
ZeroMQ and Protocol Buffers are available for a wide range
of platforms and programming languages.
Skynet
Technology Decisions
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 37
38. Skynet
Modules
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 38
skynet
skynet.network Messaging
skynet.model Data Model
skynet.process Processes
skynet.record Provenance
39. • Socket library for messaging
• Multi-platform, multi-language
• Fast and small
• Many connection patterns
ZeroMQ
www.zeromq.org
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 39
40. • Network layer based on ZeroMQ
• „Device“ as abstract endpoint
• XML configuration: ZDCF (ZeroMQ Device Configuration File)
• Automatically connected to other matching „devices“
(Peer-to-Peer)
• Compression of large data packets
• (De-)serialization via Protocol Buffer messages
• Prepared for encryption and signing
• Synchronous or asynchronous
skynet.network
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 40
41. skynet.network
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 41
42. • Specified by Google (https://github.com/google/protobuf)
• Serializing (encoding) structured data
• Efficient and extensible
• Language-neutral, platform-neutral
• Interface definition language to describe data structure
Protocol Buffers (“ProtoBuf”)
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 42
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}
43. • All models are defined as Protocol Buffer messages
• Provides basic data type as ProtoBuf (e.g., 3D vector, 6D
vector, orbital data, …)
• Interfaces for caching
• ORM for ProtoBuf messages with SQLAlchemy (database-
neutral)
• Very few additional annotation
• Compiled to Python module using protoc
• skynet.network device for database connection
skynet.model
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 43
44. If a model should be mapped to a database table, it should
have the db_table message option extension set
skynet.model
Model Definition
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 44
// Commonly used message types.
package bacardi.model.common;
import "skynet/model/proto/options.proto";
// A spatial vector with x, y and z double components
message vec3d {
required double x = 1;
required double y = 2;
required double z = 3;
}
45. skynet.model
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 45
BACARDI Network Message
HEADER BLOCK 1 BLOCK 2 BLOCK N NULL…
DEST NULL SOURCE NULL… …
TYPE COMMAND NULL [SIG]…[KEY]
46. // Orbits and orbital parameters.
package bacardi.model.orbit;
import "skynet/model/proto/options.proto";
import "bacardi/model/common.proto";
// One entry of a propagation error dataset.
message properr_item {
required double days_prop = 1;
required double altitude = 2;
required double mean_f107 = 3;
required common.vec6d std_state_rtn = 4;
}
// Propagation error dataset.
message properr {
option (skynet.model.db_table) = {
name: "properr";
};
Model Example
Orbits and Orbital Parameters
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 46
47. skynet.model
Initialize a Database with All the Models
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 47
# Import skynet ORM and database.
from skynet.model import mapping, db
# Search for models in 'bacardi.model'.
# Alternatively, you could also pass a Python module.
mapping.Mapper.scan_module('bacardi.model’)
# Connect to SQLite database
store = db.DataStore('sqlite:///bacardi.db')
# Create all tables (SQL schemas) for
# the registered models.
store.create_tables()
48. Two ways of accessing data
• Sending a Query to the DataStore
• Using the db extension
• More compact code and hence better readability
• To retrieve the ISS data, one could simply:
skynet.model
Data access
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 48
# Import data model for objects.
from bacardi.model import object_pb2
# Get the obj with NORAD ID 25544 (ISS Zarya).
iss = object_pb2.obj.get(norad_id=25544)
49. Abstraction layer for processes
• Decoupling of processes from network layer
• Automatic caching, if needed
Easy integration of processes
• as sub process (data via disk I/O or named pipes)
• as FORTRAN or C module with Python wrapper
• as pure Python module
Worker device for multiple processes
• One device per CPU / Core
skynet.process
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 50
50. Provenance is defined as a
record that describes
the people, institutions,
entities, and activities
involved in producing,
influencing, or delivering
a piece of data or a thing.
(W3C Provenance Working Group,
http://www.w3.org/2011/prov)
Provenance
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 51
Agent
Entity Activity
used
wasGeneratedBy
wasDerivedFrom
wasStartedBy
wasEndedBy
wasAssociatedWith
actedOnBehalfOf
51. Provenance recording
• Recording of all activities during runtime
• Multiple Provenance stores
• Provenance information stored as graph in Neo4j
Traceability and provability of all processed data
• Backtracking of each produced product (ephemerides,
state vectors, correlated objects, …)
• Reproducibility of products and data generated
Work-in-progress
skynet.record
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 52
52. Technical
• Graphical User Interfaces
• Web (Django) and desktop (PyQt)
• Workflow
Organizational
• Operational for simple workflows end of 2015 at DLR
• Extended SSA versions for DLR and German Army
• Other use cases for Skynet (e.g., data acquisition for
rocket motors)
BACARDI and Skynet
Current and Future Work
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 53
53. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 54
Thank You!
Questions?
Andreas.Schreiber@dlr.de
www.DLR.de/sc | @onyame
Simulated laboratory Impact of a cm - class
aluminum sphere with typical LEO-speed on
an aluminum block
The crew of the International Space Station has moved into the Soyuz vehicle docked to the station as a precaution due to an anticipated close approach of a piece of space debris to the orbiting complex. The debris is expected to pass closest to the station at about 7:01 a.m. CDT July 16, 2015. The crew will remain in the Soyuz until given an all clear by Mission Control. All station systems are currently operating normally. NASA TV will broadcast station operations live beginning at 6:45 a.m. CDT and continuing through resolution of this event. Watch NASA TV now.