The rapid development of sensing technologies had led to the creation of large volumes of environmental observation data. Data quality control information informs users how it was gathered, processed, examined. Sensor Web is a web-centric framework that involves observations from various providers. It is essential to capture quality control information within the framework to ensure that observation data are of known and documented quality. In this paper, we present a quality control framework covering different environmental observation data, and show how it is implemented in the TERENO data infrastructure. The infrastructure is modeled after the OGC’s Sensor Web Enablement (SWE) standards.
Strategies for Landing an Oracle DBA Job as a Fresher
Enabling Quality Control of SensorWeb Observations
1. Mitglied der Helmholtz-Gemeinschaft
Enabling Quality Control of
Sensor Web Observations
7th January 2014 | 3rd International Conference on Sensor Networks (SENSORNETS 2014)
Anusuriya Devaraju, Ralf Kunkel, Juergen Sorg, Heye Bogena, Harry Vereecken
4. 1. Quality Control (QC)
“…. started with activities whose purpose is to control the quality
of products or services by finding problems and defects..”1
1http://www.iso9001consultant.com.au/QA.html
The goal of QC of observation data is to identify problems
Mitglied der Helmholtz-Gemeinschaft
within the data, fixing or eliminating them, and documenting
the details involved.
4
5. 2. Sensor Web
Mitglied der Helmholtz-Gemeinschaft
Common standards
for structuring sensor
information and its
exchange.
5
13. Key Aspect of QC Information
How are data
series quality
checked? Which
quality tests are
applied?
Mitglied der Helmholtz-Gemeinschaft
What leads to
problems
within data?
Where the quality
control is
performed?
Who checks the
data?
What are the
quality levels
of the data?
When the quality
control procedure
is performed?
13
14. Research Goals
Mitglied der Helmholtz-Gemeinschaft
The goals are to capture QC information of various observation
data systematically and make the information accessible via
the Sensor Web.
14
16. Research Questions
Q1. How are raw data gathered and processed into qualitycontrolled observation data?
Mitglied der Helmholtz-Gemeinschaft
Q2. How the key aspects of data quality control can be modeled
and be related to existing observational information? How can
QC information be made available via the Sensor Web?
16
17. Mitglied der Helmholtz-Gemeinschaft
Different Ways of Importing Data
1. Data series are quality
controlled externally via
proprietary tools and then imported
into the data infrastructure
2. Data series are imported
automatically from sensors and
then quality controlled internally
(within the TEODOOR data
infrastructure).
17
18. Data Processing Status (Level)
Level
Descriptions
QC
Data Editing
Availability
Raw Data
No
No
Internal*
2a
Externally quality controlled
data; approval is pending
Yes
No, flagging only
(except human
observations)
Internal*
2b
Internally quality controlled
data with automatic QC
procedures
Yes
No, flagging only
Internal*
2c
Externally quality controlled
data with approval
Yes
No, flagging only
Public
2d
Mitglied der Helmholtz-Gemeinschaft
1
Internally quality controlled
data with combined
QC procedures (automatic
and human)
Yes
No, flagging only
Public
3
Derived data
Yes
Allowed
Public
*on request
18
19. Quality Flags (Qualifiers)
Quality Flags
GENERIC FLAGS
unevaluated
ok
baddata
suspicious
gapfilled
SPECIFIC FLAGS
moderatequality
Mitglied der Helmholtz-Gemeinschaft
goodquality
extrapolated
minerror
interpolated
badqualityquality
isolatedspike
19
20. Externally QC Data (from level 2a to 2c)
Start
Manually-uploaded, externally quality
controlled data
e.g., eddy-covariance series
fail
Send an email alert of
resubmission of data
Data importing
pass
Perform flags mapping
no
Mitglied der Helmholtz-Gemeinschaft
Processing level: Level 2a (quality controlled data without approval)
Set processing level: Level 2c (externally quality controlled data with approval)
Update approver information
Publish data via
TEODOOR
Approval
yes
End
20
21. Internally QC Data (from level 2b to 2d)
Start
Automatically-uploaded data
e.g., air temperature series
fail
Send an email alert to the responsible
scientist / field technician
DATA IMPORT
Raw data processing
pass
fail
Set processing level: Level 2b
Set generic flag: e.g., suspicious
Set specific flag: e.g., minerror (value below detection)
Automatic quality checks
Visual Inspection
Mitglied der Helmholtz-Gemeinschaft
pass
Set processing level: Level 2b
Set generic flag: ok
Set specific flag: passedautochecks
Set processing level : Level 2d (quality controlled data with automated procedures and visual inspections)
Update specific flags and evaluator information
Publish data via TEODOOR
End
21
22. Research Questions
Q1. How are raw data gathered and processed into qualitycontrolled observation data?
Mitglied der Helmholtz-Gemeinschaft
Q2. How the key aspects of data quality control can be modeled
and be related to existing observational information? How can
QC information be made available via the Sensor Web?
22
23. Observational Data Model (ODM)
sites
PK
objectid
U2,U1
code
definition
elevation_m
foi
geom
latitude
localx
localy
longitude
name
posaccuracy_m
remarks
latlondatumid
localprojectiondatumid
verticaldatumid
sources
PK
U2,U1
qualifiers
variables
objectid
PK
objectid
PK
objectid
address
administrativearea
citation
city
code
country
definition
email
firstname
link
organization
phone
surname
zipcode
metadataid
U1
code
definition
U1
U2
abbreviation
code
definition
datatypeid
offeringid
samplemediumid
timeunitid
unitid
valuetypeid
propertyid
qualifiergroups
PK
objectid
FK1
FK2
groupid
qualifierid
processingstati
PK
PK
U1
code
definition
link
manufacturer
model
type
version
terenodata
objectid
FK1
FK7
FK3
I1
FK4
methods
objectid
U1
Mitglied der Helmholtz-Gemeinschaft
PK
code
definition
link
organization
FK6
I2
FK5
FK2
objectid
U1
sensors
code
definition
shortdesc
U2
timestampto
processingstatusid
siteid
variableid
The existing observational data model
has been modified to support quality
control descriptions
• Qualifiers (quality flags)
• Data processing status
• Source
• Method..etc.
objectid
datavalue
datavalueaccuracy
offsetvalue
timestampfrom
censorcodeid
importid
methodid
offsettypeid
qualifierid
sampleid
sourceid
validationsourceid
derivedfrom
binobject
binobjecttypeid
usersitevariablepermissions
PK
objectid
U1
U1
FK1,U1
groupsetid
siteid
sourceid
variableid
loggervariables
PK
sensorcomponents
PK
objectid
U1
code
definition
functionid
methodid
sensorid
sensortypeid
FK1,U1
FK2,U1
U1
FK1,U1
FK3
FK4,U1
FK2,U1
U1
logger
objectid
PK
objectid
allowedmaxvalue
allowedminvalue
importfactor
loggerfilecolumnname
loggerfilecolumnnumber
loggerid
processingstatusid
sampletypeid
sensorcomponentid
variableid
sensorinstanceid
U1
code
definition
technicalwarningdays
timestampfrom
timestampto
datatableclassid
filetypeid
sourceid
timezone
siteid
notify
U1
23
24. QC-Enabled SOS
Quality Flags
Observation Values
Mitglied der Helmholtz-Gemeinschaft
Data Processing Status
Each value is accompanied with a reference
combining quality flag id and data processing
24
status id
28. Summary
A common quality control framework for processing and assessing
time series from various sensing applications of TERENO
infrastructure. The framework consists of:
A common QC workflow covering various sensor data
•
An extensible quality flag classification
•
Changes applied to existing observational data model
•
QC-Enabled SOS
•
Sensor Web Client(s) delivering quality controlled observation
data.
Mitglied der Helmholtz-Gemeinschaft
•
28
29. What’s Next?
Extend the
observation request
of the SOS with QCbased filters
1.
Mitglied der Helmholtz-Gemeinschaft
1.
Incorporate
descriptions about
operation and
maintenance
sensing systems in
the Sensor Web
29
30. Thank you.
Mitglied der Helmholtz-Gemeinschaft
For more information, please visit:
http://teodoor.icg.kfa-juelich.de
30