Presented at the 3rd ICOS Science Conference in Prague, Czech Republic, 11-13 Sep 2018.

1. The AmeriFlux Network Data
Management System
Deb Agarwal, Gilberto Pastorello, Danielle
Christianson, You-Wei Cheah, , Housen Chu
Sebastien Biraud, Margaret Torn
Berkeley Lab, Berkeley, CA, USA

2. Map of
AmeriFlux Sites
0
10
20
30
40
50
2013 2014 2015 2016 2017 2018
# Sites Joined AmeriFlux
March 2013 = 154 sites
September 2018 = 348 sites

3. -10
10
30
50
70
90
110
130
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
AmeriFlux Site Years Availability
Site Years Published 2013
Site Years Published 2013-2018
Site Years Pending QA/QC
NumberofSiteYearsAvailable
Year

4. AmeriFlux Management Project
(2013 - ?)
• AmeriFlux web site
• Core site funding (~40 sites)
• Site QA/QC visits
• Community coordination
• Data Team
– Data standardization – BADM and FP
– Data services including upload, search, and download
– Data QA/QC (since Sept 2016)
– FLUXNET data support since LaThuile (collaborative)

5. Format
QA/QC
Data
QA/QC
BASE
Publish
Does file format
comply with
standard?
Does data pass
quality checks?
BASE Data
Product available
to network
Tower Team
submission
AmeriFlux Data Processing Pipeline
• BASE = the standard AmeriFlux data product consisting of half-
hourly / hourly time series in FP-Standard

6. Uploaded
File NOT
Considered
for BASE
File
AutoRepaired
Format Fixer
File NOT
AutoRepaired
Format
Checker
File Ready
for BASE
AutoRepaired File
Original File
Automated Format QA/QC
FP-In Standard:
• Filename
• Variable names
• Timestamps
• Temporal Resolution
• Missing value format
• Missing data
• Gap-filled

7. QA/QC Report Tracking and Communications Tracking

8. Data QA/QC is semi-automated
File Ready
for BASE
Combiner
Data QAQC
Checker
Curation
BASE
Attempt
Failed Proposed
BASE
BASE
Generation
BASE–BADM
Generation
Published
BASE--BADM
Data QA/QC check for:
• Units, Sign convention
• Outliers, physical range
• Diurnal & Seasonal pattern
• Multivariate comparison
• Time shift check
• USTAR-filtering

9. Data QA/QC Report and JIRA for site level analysis
10/15

10. Quality Checks Per Month
0
20
40
60
80
100
120
140
Jul -
2017
Aug -
2017
Sep -
2017
Oct -
2017
Nov -
2017
Dec -
2017
Jan -
2018
Feb -
2018
Mar -
2018
Apr -
2018
May -
2018
Format QA/QC
Data QA/QC
NumberofSitesChecked

11. 0
2000
4000
6000
8000
10000
12000
14000
AmeriFlux - Number of
Downloads Per Month
~4700 Downloads/month (1875 unique users)
DOIs for all published data
NumberofSiteDownloads

12. FLUXNET Datasets
• Marconi 2000: 38 sites / 97 site-years
– Data contributed by attendees of workshop
– Access limited to attendees
• LaThuile 2007: 252 sites / 965 site-years
– Data contributed by many site PIs
– Start of uniform code base and processing for comparable data products
– Data access/usage opened to other users, with some restrictions (three tiers)
– Over 500 users (as of Sep 7, 2018)
• FLUXNET2015: 212 sites / 1532 site-years
– Standardized code base (developed in collaboration among networks)
– Extensive QA/QC of data (data more comparable and usable, but fewer sites)
– Data access to anyone, usage has fewer restrictions (two tiers) – tools to keep track of usage
– Over 1600 users (as of Sep 7, 2018)

13. 0
5000
10000
15000
20000
25000
30000
FLUXNET2015 Downloads By Month
and Region
AmeriFlux ICOS Europe - ICOS Asia Ozflux Other
1203 Unique Users
DOIs for all
published data
NumberofSiteDownloads
~4700/mo ~2500/mo ~2900/mo ~1000/mo ~1600/mo ~800/mo

14. AmeriFlux BADM Roadmap
• Existing structure and technology was built to support AmeriFlux and LaThuile
– Limited number of groups
– No existing standard for submission
• Next-gen infrastructure
– Incorporate experience
– Support more rapid testing and development of new BADM groups
– Streamline deployment, submission, web updates, and processing
– VegCover starting this fall
• Near future targets
– Veg Cover
– DOI enhancements
– Biomass
• Longer term goals:
– New collection mechanisms: Template revamp, Web collection

15. ONEFlux Processing Pipeline:
Open Source collaborative code
Gilberto Pastorello, Carlo Trotta, Alessio
Ribeca, Dario Papale, Abdelrahman
Elbashandy, Alan Barr
Markus Reichstein, Gitta Lasslop, Deb
Agarwal, Sebastien Biraud, et al.
ONEFlux coding team
Gilberto
Pastorello
Abdelrahman
Elbashandy
Carlo
Trotta
Alessio
Ribeca
Dario
Papale
Alan
Barr

16. ONEFlux: Software Pipeline
https://github.com/AmeriFlux/ONEFlux
ONEFlux Pipeline
• Shared code repository
– ~70k lines of code (v0.3.0-beta) – primarily python
– BSD license
• Shared documentation
• Execution and testing improvement
– From supervised & manual execution - days to
weeks to run a site
– To automated pipeline - average 4h per site-year
• Validation and testing protocols
– ~280k lines for validation, testing, and comparisons
– Running FLUXNET2015 to check (75% complete)

17. Next Steps
• Expand and improve available AmeriFlux flux/met data
• Expand and improve AmeriFlux BADM collection
• Jupyter Notebooks for using BADM data
• ESS-DIVE Data archiving
• ONEFlux next steps
• FLUXNET next steps
– Replacement for all sites map (Max Planck Inst – Jena, University of Tuscia,
AmeriFlux)
– Next product?

18. Thank you
• Regional Networks – past, present, and future
• Site PIs
• ICOS Ecosystem Thematic Center
• AMP Project Team
• DOE
• Max Planck Institute Jena

19. 0
20
40
60
80
100
120
140
NUMBEROFSITES
SITE YEARS
AmeriFlux Site Years Availability
Site Years Published
Site Years Pending QA/QC

20. 0
5000
10000
15000
20000
25000
30000
35000
12/1/15
1/1/16
2/1/16
3/1/16
4/1/16
5/1/16
6/1/16
7/1/16
8/1/16
9/1/16
10/1/16
11/1/16
12/1/16
1/1/17
2/1/17
3/1/17
4/1/17
5/1/17
6/1/17
7/1/17
8/1/17
9/1/17
10/1/17
11/1/17
12/1/17
1/1/18
2/1/18
3/1/18
4/1/18
5/1/18
6/1/18
7/1/18
8/1/18
FLUXNET2015 Downloads by Tier
Tier1 Tier2