Public Sector Summit Insights

P U B L I C S E C T O R
S U M M I T
WASH INGTON, DC

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Seeing the stars through the
cloud
3 0 1 0 9 5
Aisen Etcheverry Escudero
DO Interim Director
Data Observatory for Government of Chile
Alex Coqueiro
Head of Solutions Architecture Team
AWS Latin America, Canada & Caribbean
@alexbcbr

S U M M I T

S U M M I T
Agenda
WHY Government of Chile is investing in stargazing
HOW we implemented this project

Chile - is a very peculiar country
Source: Marca Chile

1847 1854 1927 1959 1963 1996 1997 19981961 2006 2010 2013 2017 2019 2020+
S A L I T R E C O P P E R
MI N I N G C O N S TR U C TI O N
C I TI E S
A G R I C U L TU R E
E N V I R O N ME N T
H E A L TH
K N O W L E D G E
WITH A VERY PECULIAR HISTORY

1849 1854 1927 1959
James
T. Gilliss
Observatorio
Astronómico
Nacional
(OAN).
Cerro
Calán
Radiotelescopio
de Maipú
1963 1996 1997 19981961
10%
CTIO
La Silla
Paranal
GEMINI
Protección
Lumínica
2006 2010 2013 2017 2019 2020+
C a n c i l l e r í a
D E C I T Y
A L M A
P r o g r a ma
A s t r o i n f o r má t i c a
M I N E C O N
P a r q u e
A s t r o n ó mi c o
d e A t a c a ma
D O
L S S T
E - E L T
C C A T p
C T A
G M T
B E G I N N I N G S M E G A - T E L E S C O P E S E R A L A S T D E C A D E
WITH A VERY PECULIAR HISTORY

S U M M I T
….and its role in astronomy is key

Astronomy is among the fields
producing more exploitable data!

S U M M I T
This data has large volume, variety, velocity, and
veracity
C T A
V L T
G M T
L S S T
A L M A
S I M O N S
G E M I N IN T T
L C O
A C T
A P E X L C T
C C A T P
~ 2 0 P B P E R Y E A R
M I L K Y W A Y I N M U L T I P L E W A V E L E N G T H S
S O A R E E L T

S U M M I T
This data is relevant for science
Eagle Simulation, 2016
C T A
V L T
G M T
L S S T
A L M A
S I M O N SG E M I N IN T T
L C O
A C T
A P E X L C T
C C A T P
S O A R E E L T
How did the universe form and grow?

S U M M I T
Drake, 1961
Are we alone in the Universe?
This data is relevant for humanity
C T A
V L T
G M T
L S S T
A L M A
S I M O N SG E M I N IN T T
L C O
A C T
A P E X L C T
C C A T P
S O A R E E L T

S U M M I T
A s t r o n o m y i s 5 -
1 0 y e a r s a h e a d o f
o t h e r a c t i v i t i e s i n
t h e r e g i o n
This data is relevant for economic development

S U M M I T
DOING AS OTHER ACTIVITIES,
5 TO 10 YEARS AHEAD
≈
HEALTHCARE
CLIMATE
CHANGE
SCADA
systems
PRECISION
AGRICULTURE
SMARTCITIES
COMMERCE
An engine of
digital transformation
≈

S U M M I T
1st, we gathered the data…

S U M M I T
First Digital
Astronomy (~1970)
Full Digital
Astronomy (~1990)
Virtual Observatory
(~2010)

S U M M I T
Cloud-Based Astronomy
(Data Observatory, ~2020)

S U M M I T
Cloud-Based Astronomy
(Data Observatory, ~2020)
Bring all computing, including
data access, visualisation, and
scientific collaboration to the data

S U M M I T
2nd, we secured government support and long-term commitment

S U M M I T
3rd, we use technology and partner with the right people

S U M M I T
The results so far….
We generated a collaboration that resulted in a framework for predictive
maintenance of ALMA
Source: NRAO

S U M M I T
The Atacama Large Millimeter/submillimeter
Array (ALMA) is a radio telescope made up of 66
antennas which can work together as a single
antenna with a total diameter of 16 kilometers.
What is ALMA?

S U M M I T
We created a mechanism for automatic processing of ALMA images on the cloud
HL Tauri
Real ALMA image (ESO/NAOJ/NRAO)

S U M M I T
HL Tauri
Without Self-Calibration (simulated)

S U M M I T
HL Tauri
With Self-Calibration
Manual process for self-
calibration takes at least
2h, typically 4h to 24h
Machine Learning
model for self-calibration
training in order to bring
execution down to
minutes.

S U M M I T
We developed a prototype for visualization of ALMA data

S U M M I T
Vision
The DO is at the vanguard of data-centric
innovation, leading in the production of data-centric
solutions, talent, and social capital for the Latin-
American region.
Mission
The DO hosts datasets of global value acquired and
generated in our region, and enables their maximal
exploitation by the global scientific community, the
industry and the public, facilitating data access,
analysis, exploration, visualization, and governance.
Finally, we created a permanent collaboration for much more…

S U M M I T
Where do we want to get….
MI N E R I A C O N S TR U
C I U D A D E S
A G R O
C L I MA
S A L U D

S U M M I T How did you do it?

S U M M I T
… Architecture - Big Picture …

S U M M I T
DO project is a data-centric approach for
astronomy
Designed by Demian Arancibia

S U M M I T
Approach for data acquisition
Online
data transfer
Hybrid
storage
Offline
data transfer
Private
network
connections
to AWS
Load
streaming
data into
Amazon S3
Ship static
data into
and out of
Amazon S3
Access AWS
storage from
on premises
Edge
locations for
simple
Storage
Service
(Amazon S3)-
enabled
applications
Online
transfer of
active data
AWS DataSync AWS Transfer
for SFTP
SFTP transfers
into Amazon
S3
Storage and
compute in
disconnected
environment
s

S U M M I T
• 100-TB storage capacity
• Data encryption end-to-
end
• Rugged 8.5-G impact case
• Rain and dust resistant
• AWS IoT Greengrass
support for local compute,
messaging, and caching
• Amazon Elastic Compute
Cloud (Amazon EC2)/AMI
support for edge compute
AWS Snowball Edge Storage Optimized

S U M M I T
Curated Datasets of Global Value and Relevance

S U M M I T
Data as a Product - Open Data as a Platform
Data Creation Data Enrichment
Sensemaking
Data at Rest
(Object storage)
Basic APIs
Complex APIs
Consumer
applications
Algorithmic
policy
Data-driven
journalism
Data Catalogs
Focused data
dashboards
Predictive
modeling
Visualizations
Lower cost of knowledge
(Efficiency)

Views of Centaurus A galaxy in different wavelengths.View of Centaurus A galaxy view in millimeter and submillimeter radio wavelength from ALMA radio telescope.
Credit: ALMA (ESO/NAOJ/NRAO).
Undestanding Image Capturing Process
Scientific technological development today
allows scientists to design equipment that can
capture a wider range of wavelengths.
Optical telescopes, which observe the light
of the stars and hot gas emission.
Ultraviolet and X-Ray observatories that
study very hot gas and ultra energetic
emission.
Infrared light observatories that study warm
and hot dust.
Radio telescopes that capture radio
emissions from warm and cold matter.
ALMA observes millimetric and
submillimetric radio wavelengths.
View of Centaurus A galaxy in visible light wavelength from La Silla telescope.
Credit: La Silla observatory.
View of Centaurus A galaxy in X-ray wavelength from Chandra observatory.
Credit: Chandra observatory.
View of Centaurus A galaxy in infrared wavelength from Spitzer observatory.
Credit: Spitzer observatory.
Combined view of Centaurus A galaxy in visible light and X-ray wavelengths.View of Centaurus A galaxy in radio wavelength from VLA radio telescope.
Credit: VLA radio telescope.

What ALMA can see …
By observing millimetric wavelengths, ALMA
can see the so called “Cold Universe”,
where planets and galaxies are formed.
ALMA has an unprecedented capability for
discovering the presence and distribution
of molecules in space and can observe,
with high resolution, masses of cold and
warm gas around protoplanets.
By observing the "invisible" ALMA can see
our sun in detail and capture light emitted
by particles rather than light reflected by
them.
Thus, ALMA opens up innumerable
possibilities for astronomical research,
from the discovery of new interstellar
molecules to the mysteries of our cosmic
origins.
Glycolaldehyde molecules in the gas surrounding the star known as IRAS 16293-2422, 2012.
Combined view of the dust ring around the bright star Fomalhaut from the ALMA observatory and Hubble telescope, 2012.
Credit: ALMA (ESO/NAOJ/NRAO) / NASA / ESA.
Spiral structure of the material around the old star R Sculptoris revealed by ALMA, 2012.

S U M M I T
Changing the Traditional Approach
Data is most commonly accessed
via a web interface (archive) and
downloaded on premises before
being analysed.
Files are downloaded in a .tar
archive.
Data acquisition is time consuming
and inherently redundant.

Data lake in the cloud
Amazon S3

S U M M I T
Logical Dataflow Design
Designed by Dr. Juande Santander
D A T A L A K E
B O U N D A R Y

S U M M I T
We have the foundation … so…

S U M M I T
AWS Cloud
High-Throughput Radio Astronomy (Andrés Fernández-León Thesis )
Simulation
Self Calibration
DO Processing
Tensorflow
Conda/CUDA Message
DO Datalake
FSx for
Lustre
Conda
CUDA
Sample Generation
Simulator

S U M M I T
DO Harvester
AWS Cloud
Astronomy 4.0 (José Ortiz and Rodrigo Carrasco)
Metadata
DO Datalake
Inference Maintenance
Prediction
Internet
Internal
Telescope
Antenna
Correlator
Maintenance
Team
HDF5
Table
Consumers
DO Front End
Website
pages
Multi-model system based on
Kalman filters identifying
intermediate frequency
processors amplifiers
degradation

S U M M I T
Protect Detect Respond
Automate
Investigate
RecoverIdentify
Snapshot
NIST Cybersecurity Framework
VPN Connection

S U M M I T
DO Ingest
DO Data Store DO Data Visualizer
DO Processing
DO Data Harvester
Aladin
Data Observatory
Discovery
FITS
HIPSRegistry

S U M M I T
as a product
=>

S U M M I T
Thank YOU & Gracias
3 0 1 0 9 5
Aisen Etcheverry Escudero
DO Interim Director
Data Observatory for Government of Chile
@alexbcbr
Alex Coqueiro
Head of Solutions Architecture Team
AWS Latin America, Canada & Caribbean

S U M M I T
S U M M I T
Please complete the
session survey.
!

Public Sector Summit Insights

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Public Sector Summit Insights

Ähnlich wie Public Sector Summit Insights (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

Public Sector Summit Insights