SlideShare a Scribd company logo
1 of 13
Download to read offline
XLDB CONFERENCE 2015, STANFORD UNIVERSITY MAY 2015
How To Create the Google for Earth Data
Rainer Sternfeld
CEO & co-founder
in the example of NOAA Big Data Project
2 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
Finding and accessing the right data is really hard
Planet OS Data Discovery makes it easy
Crawl the web and index the most recent data
without moving the data itself until you need it
3 August 2014
Cloud Platform for Industrial Sensor Networks
Building with:
Scala, Akka
RabbitMQ, Kafka
HDFS, HBase, Elasticsearch
Spark, Spark Streaming
GIS Libraries (raster, geometry)

Compressed array formats
Sensor Data Discovery & Exchange
Search, Exploration
Visualisation, Analytics
Data Management
Marketplace
APIs
4 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
marinexplore.org — the biggest deployment of Planet OS
43,000+ data streams, 35 organizations, 8,000 users
Data Discovery Raster data, heatmap overlays Access with third party applications
Raster data with quiver plot overlays Rich graph visualizations Build custom datasets
5 April 2015
Marine data website
Visualising
Browsing
Filtering
Export
Built with:
Python/Cython
RabbitMQ, Celery
Postgres, PostGIS, Vertica

GIS Libraries (raster, geometry)
NumPy
marinexplore.org
6 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
How to make NOAA’s large-scale weather and climate data
easily discoverable and machine-readable?
Real-time weather & climate data is
a global multi-billion dollar opportunity
7 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
NOAA’s Challenge
• Tens of thousands of devices deployed in the ocean, on land, and space
• Critical for the government and industries
• Hundreds scattered web services (FTPs, flat files, THREDDS/OPeNDAP API)
• Data grows 10TB+ per day
• 26,595 NOAA datasets with ISO-19139 metadata
• 4,894 NOAA datasets with OpenDAP interface
8 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
So what’s wrong with how it’s done now?
An enthusiastic young researcher starts downloading data to an
external HDD connected to his laptop — data keeps coming,
external HDDs pile up…


the beard’s getting longer and longer, and when the data is
finally downloaded, we have a middle-aged, bored professor
9 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
Technical Challenges in
the NOAA Big Data Project
• Storing, processing, and indexing spatio-temporal time-series & array data
• Processing data at 10s (100s) of TB/day
• Transporting and processing archives at volumes 10s (100s) of PB
• Disseminating real-time data at latency of minutes
• Indexing 100K+ logical datasets and 100M+ technical datasets/files
• Providing uniform API/export for various data formats/protocols/projections
10 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
Potential solutions
under consideration
• Indexing spatio-temporal and semantic metadata
• Indexing downsampled remote datasets acquired via OpenDAP and others
• Store chunked array data (MBs) in object store (e.g. Amazon S3)
• Provide on-demand computational infrastructure for analyzing data (e.g. Amazon)
11 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
What would make it even better?
• Incremental data compression
• BitTorrent-like data dissemination
• Sending pre-filtered data to consumers (e.g. by area)
• Computations scheduled next to the data storage
• Fast interconnect (10 GBit / Inifiniband) and GPUs
• Run analytical scripts (eg IPython Notebook, Matlab) to
work with array data in the cloud
12 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY
DATA
EXCHANGE
APIsPLANET OS
ALGORITHMS
3RD PARTY
INTEGRATIONS
DISCOVER
COLLABORATE
VISUALIZE
ENTERPRISE
DATA
ALL PUBLIC DATA
MODELS APPS
ANALYZE
We index your world

More Related Content

What's hot

SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...
SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...
SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...
OpenTopography Facility
 
Big Data Is Not Enough - Real-Time Analytics Needs Streaming Archtectures
Big Data Is Not Enough - Real-Time Analytics Needs Streaming ArchtecturesBig Data Is Not Enough - Real-Time Analytics Needs Streaming Archtectures
Big Data Is Not Enough - Real-Time Analytics Needs Streaming Archtectures
Dr. Tim Frey
 

What's hot (17)

OVH Analytics Data Compute and Apache Spark as a Service
OVH Analytics Data Compute and Apache Spark as a ServiceOVH Analytics Data Compute and Apache Spark as a Service
OVH Analytics Data Compute and Apache Spark as a Service
 
Super BDaaS Government
Super BDaaS GovernmentSuper BDaaS Government
Super BDaaS Government
 
Big Data - part 5/7 of "7 modern trends that every IT Pro should know about"
Big Data - part 5/7 of "7 modern trends that every IT Pro should know about"Big Data - part 5/7 of "7 modern trends that every IT Pro should know about"
Big Data - part 5/7 of "7 modern trends that every IT Pro should know about"
 
Airline traffic management analysis
Airline traffic management analysisAirline traffic management analysis
Airline traffic management analysis
 
Research Data Australia and the national research data landscape
Research Data Australia and the national research data landscapeResearch Data Australia and the national research data landscape
Research Data Australia and the national research data landscape
 
Experience Big Data Analytics use cases ranging from cancer research to IoT a...
Experience Big Data Analytics use cases ranging from cancer research to IoT a...Experience Big Data Analytics use cases ranging from cancer research to IoT a...
Experience Big Data Analytics use cases ranging from cancer research to IoT a...
 
Chek mate geolocation analyzer
Chek mate geolocation analyzerChek mate geolocation analyzer
Chek mate geolocation analyzer
 
Geolocation analysis using HiveQL
Geolocation analysis using HiveQLGeolocation analysis using HiveQL
Geolocation analysis using HiveQL
 
PIOTRe: Personal Internet of Things Repository
PIOTRe: Personal Internet of Things RepositoryPIOTRe: Personal Internet of Things Repository
PIOTRe: Personal Internet of Things Repository
 
BUILDING BETTER PREDICTIVE MODELS WITH COGNITIVE ASSISTANCE IN A DATA SCIENCE...
BUILDING BETTER PREDICTIVE MODELS WITH COGNITIVE ASSISTANCE IN A DATA SCIENCE...BUILDING BETTER PREDICTIVE MODELS WITH COGNITIVE ASSISTANCE IN A DATA SCIENCE...
BUILDING BETTER PREDICTIVE MODELS WITH COGNITIVE ASSISTANCE IN A DATA SCIENCE...
 
SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...
SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...
SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...
 
Predicting Loan Delinquency at One Million Transactions per Second
Predicting Loan Delinquency at One Million Transactions per SecondPredicting Loan Delinquency at One Million Transactions per Second
Predicting Loan Delinquency at One Million Transactions per Second
 
Distributed system
Distributed systemDistributed system
Distributed system
 
CouchDB Day NYC 2017: Using Geospatial Data in Cloudant & CouchDB
CouchDB Day NYC 2017: Using Geospatial Data in Cloudant & CouchDBCouchDB Day NYC 2017: Using Geospatial Data in Cloudant & CouchDB
CouchDB Day NYC 2017: Using Geospatial Data in Cloudant & CouchDB
 
Floods of Twitter Data - StampedeCon 2016
Floods of Twitter Data - StampedeCon 2016Floods of Twitter Data - StampedeCon 2016
Floods of Twitter Data - StampedeCon 2016
 
Big Data Is Not Enough - Real-Time Analytics Needs Streaming Archtectures
Big Data Is Not Enough - Real-Time Analytics Needs Streaming ArchtecturesBig Data Is Not Enough - Real-Time Analytics Needs Streaming Archtectures
Big Data Is Not Enough - Real-Time Analytics Needs Streaming Archtectures
 
PowerStream: Propelling Energy Innovation with Predictive Analytics
PowerStream: Propelling Energy Innovation with Predictive Analytics PowerStream: Propelling Energy Innovation with Predictive Analytics
PowerStream: Propelling Energy Innovation with Predictive Analytics
 

Similar to How to Create the Google for Earth Data (XLDB 2015, Stanford)

So Long Computer Overlords
So Long Computer OverlordsSo Long Computer Overlords
So Long Computer Overlords
Ian Foster
 
INF2190_W1_2016_public
INF2190_W1_2016_publicINF2190_W1_2016_public
INF2190_W1_2016_public
Attila Barta
 
Rpi talk foster september 2011
Rpi talk foster september 2011Rpi talk foster september 2011
Rpi talk foster september 2011
Ian Foster
 

Similar to How to Create the Google for Earth Data (XLDB 2015, Stanford) (20)

Scaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data ScienceScaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data Science
 
Spark at NASA/JPL-(Chris Mattmann, NASA/JPL)
Spark at NASA/JPL-(Chris Mattmann, NASA/JPL)Spark at NASA/JPL-(Chris Mattmann, NASA/JPL)
Spark at NASA/JPL-(Chris Mattmann, NASA/JPL)
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data Challenges
 
Hopsworks in the cloud Berlin Buzzwords 2019
Hopsworks in the cloud Berlin Buzzwords 2019 Hopsworks in the cloud Berlin Buzzwords 2019
Hopsworks in the cloud Berlin Buzzwords 2019
 
AMP Camp 5 Intro
AMP Camp 5 IntroAMP Camp 5 Intro
AMP Camp 5 Intro
 
So Long Computer Overlords
So Long Computer OverlordsSo Long Computer Overlords
So Long Computer Overlords
 
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and AutomationThe Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
 
Louise McCluskey, Kx Engineer at Kx Systems
Louise McCluskey, Kx Engineer at Kx SystemsLouise McCluskey, Kx Engineer at Kx Systems
Louise McCluskey, Kx Engineer at Kx Systems
 
INF2190_W1_2016_public
INF2190_W1_2016_publicINF2190_W1_2016_public
INF2190_W1_2016_public
 
Using the Open Science Data Cloud for Data Science Research
Using the Open Science Data Cloud for Data Science ResearchUsing the Open Science Data Cloud for Data Science Research
Using the Open Science Data Cloud for Data Science Research
 
Big Data Session 1.pptx
Big Data Session 1.pptxBig Data Session 1.pptx
Big Data Session 1.pptx
 
Rpi talk foster september 2011
Rpi talk foster september 2011Rpi talk foster september 2011
Rpi talk foster september 2011
 
President Election of Korea in 2017
President Election of Korea in 2017President Election of Korea in 2017
President Election of Korea in 2017
 
Time to Science/Time to Results: Transforming Research in the Cloud
Time to Science/Time to Results: Transforming Research in the CloudTime to Science/Time to Results: Transforming Research in the Cloud
Time to Science/Time to Results: Transforming Research in the Cloud
 
eScience: A Transformed Scientific Method
eScience: A Transformed Scientific MethodeScience: A Transformed Scientific Method
eScience: A Transformed Scientific Method
 
160606 data lifecycle project outline
160606 data lifecycle project outline160606 data lifecycle project outline
160606 data lifecycle project outline
 
Minimizing the Complexities of Machine Learning with Data Virtualization
Minimizing the Complexities of Machine Learning with Data VirtualizationMinimizing the Complexities of Machine Learning with Data Virtualization
Minimizing the Complexities of Machine Learning with Data Virtualization
 
Accelerating Science with Cloud Technologies in the ABoVE Science Cloud
Accelerating Science with Cloud Technologies in the ABoVE Science CloudAccelerating Science with Cloud Technologies in the ABoVE Science Cloud
Accelerating Science with Cloud Technologies in the ABoVE Science Cloud
 
What is a Data Commons and Why Should You Care?
What is a Data Commons and Why Should You Care? What is a Data Commons and Why Should You Care?
What is a Data Commons and Why Should You Care?
 
Data Science with the Help of Metadata
Data Science with the Help of MetadataData Science with the Help of Metadata
Data Science with the Help of Metadata
 

Recently uploaded

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
amitlee9823
 
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
amitlee9823
 
➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men 🔝mahisagar🔝 Esc...
➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men  🔝mahisagar🔝   Esc...➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men  🔝mahisagar🔝   Esc...
➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men 🔝mahisagar🔝 Esc...
amitlee9823
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
amitlee9823
 
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
amitlee9823
 
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
amitlee9823
 
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
amitlee9823
 
Probability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter LessonsProbability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter Lessons
JoseMangaJr1
 
Call Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night StandCall Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night Stand
amitlee9823
 
➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...
➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...
➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...
amitlee9823
 
➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...
➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...
➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...
amitlee9823
 
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
amitlee9823
 
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service BangaloreCall Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
amitlee9823
 
➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...
➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...
➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...
amitlee9823
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
amitlee9823
 

Recently uploaded (20)

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
 
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
 
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
 
➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men 🔝mahisagar🔝 Esc...
➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men  🔝mahisagar🔝   Esc...➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men  🔝mahisagar🔝   Esc...
➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men 🔝mahisagar🔝 Esc...
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
 
Thane Call Girls 7091864438 Call Girls in Thane Escort service book now -
Thane Call Girls 7091864438 Call Girls in Thane Escort service book now -Thane Call Girls 7091864438 Call Girls in Thane Escort service book now -
Thane Call Girls 7091864438 Call Girls in Thane Escort service book now -
 
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
 
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
 
Capstone Project on IBM Data Analytics Program
Capstone Project on IBM Data Analytics ProgramCapstone Project on IBM Data Analytics Program
Capstone Project on IBM Data Analytics Program
 
5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed
5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed
5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed
 
Probability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter LessonsProbability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter Lessons
 
Call Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night StandCall Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Doddaballapur Road ☎ 7737669865 🥵 Book Your One night Stand
 
➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...
➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...
➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...
 
➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...
➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...
➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...
 
hybrid Seed Production In Chilli & Capsicum.pptx
hybrid Seed Production In Chilli & Capsicum.pptxhybrid Seed Production In Chilli & Capsicum.pptx
hybrid Seed Production In Chilli & Capsicum.pptx
 
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
 
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service BangaloreCall Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
 
➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...
➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...
➥🔝 7737669865 🔝▻ Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
 

How to Create the Google for Earth Data (XLDB 2015, Stanford)

  • 1. XLDB CONFERENCE 2015, STANFORD UNIVERSITY MAY 2015 How To Create the Google for Earth Data Rainer Sternfeld CEO & co-founder in the example of NOAA Big Data Project
  • 2. 2 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY Finding and accessing the right data is really hard Planet OS Data Discovery makes it easy Crawl the web and index the most recent data without moving the data itself until you need it
  • 3. 3 August 2014 Cloud Platform for Industrial Sensor Networks Building with: Scala, Akka RabbitMQ, Kafka HDFS, HBase, Elasticsearch Spark, Spark Streaming GIS Libraries (raster, geometry)
 Compressed array formats Sensor Data Discovery & Exchange Search, Exploration Visualisation, Analytics Data Management Marketplace APIs
  • 4. 4 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY marinexplore.org — the biggest deployment of Planet OS 43,000+ data streams, 35 organizations, 8,000 users Data Discovery Raster data, heatmap overlays Access with third party applications Raster data with quiver plot overlays Rich graph visualizations Build custom datasets
  • 5. 5 April 2015 Marine data website Visualising Browsing Filtering Export Built with: Python/Cython RabbitMQ, Celery Postgres, PostGIS, Vertica
 GIS Libraries (raster, geometry) NumPy marinexplore.org
  • 6. 6 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY How to make NOAA’s large-scale weather and climate data easily discoverable and machine-readable? Real-time weather & climate data is a global multi-billion dollar opportunity
  • 7. 7 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY NOAA’s Challenge • Tens of thousands of devices deployed in the ocean, on land, and space • Critical for the government and industries • Hundreds scattered web services (FTPs, flat files, THREDDS/OPeNDAP API) • Data grows 10TB+ per day • 26,595 NOAA datasets with ISO-19139 metadata • 4,894 NOAA datasets with OpenDAP interface
  • 8. 8 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY So what’s wrong with how it’s done now? An enthusiastic young researcher starts downloading data to an external HDD connected to his laptop — data keeps coming, external HDDs pile up… 
 the beard’s getting longer and longer, and when the data is finally downloaded, we have a middle-aged, bored professor
  • 9. 9 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY Technical Challenges in the NOAA Big Data Project • Storing, processing, and indexing spatio-temporal time-series & array data • Processing data at 10s (100s) of TB/day • Transporting and processing archives at volumes 10s (100s) of PB • Disseminating real-time data at latency of minutes • Indexing 100K+ logical datasets and 100M+ technical datasets/files • Providing uniform API/export for various data formats/protocols/projections
  • 10. 10 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY Potential solutions under consideration • Indexing spatio-temporal and semantic metadata • Indexing downsampled remote datasets acquired via OpenDAP and others • Store chunked array data (MBs) in object store (e.g. Amazon S3) • Provide on-demand computational infrastructure for analyzing data (e.g. Amazon)
  • 11. 11 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY What would make it even better? • Incremental data compression • BitTorrent-like data dissemination • Sending pre-filtered data to consumers (e.g. by area) • Computations scheduled next to the data storage • Fast interconnect (10 GBit / Inifiniband) and GPUs • Run analytical scripts (eg IPython Notebook, Matlab) to work with array data in the cloud
  • 12. 12 MAY 2015XLDB CONFERENCE 2015, STANFORD UNIVERSITY DATA EXCHANGE APIsPLANET OS ALGORITHMS 3RD PARTY INTEGRATIONS DISCOVER COLLABORATE VISUALIZE ENTERPRISE DATA ALL PUBLIC DATA MODELS APPS ANALYZE
  • 13. We index your world