Presented at the IMAODBC - Neuchatel, 16-20 September 2013

1. The OECD Delta Project
IMAODBC - Neuchatel, 16-20 September 2013
Jonathan Challener, OECD
{“Providing easier access to data”:“Through APIs”}

2. Accessible
Open
Free
Find
Understand
Use
Machine-readable
Indexable
Re-Useable
Available without
charge
Making OECD data Open, Accessible, Free

3. Accessible
Open
Free
Find
Understand
Use
Machine-readable
Indexable
Re-Useable
Available without
charge
Making OECD data Open, Accessible, Free

4. {“the oecd experience”:“2007-2012”}

5. Machine-readable
SDMX is the global standard for statistical
data and metadata exchange

6. SDMX-ML 2.0 Web Service
<?xml version="1.0"?>
<message:MessageGroup
xmlns:message="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/generic
http://www.sdmx.org/docs/2_0/SDMXGenericData.xsd
http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message
http://www.sdmx.org/docs/2_0/SDMXMessage.xsd"
xmlns:common="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/common"
xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/generic"><Header
xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message">
<ID>none</ID><Test>false</Test><Truncated>false</Truncated><Prepared>2013-04-
15T11:46:19</Prepared>
<Sender id="OECD"><Name xml:lang="en">Organisation for Economic Co-operation and
Development</Name>
<Name xml:lang="fr">Organisation de coopération et de développement
économiques</Name></Sender></Header>
<DataSet
keyFamilyURI="http://stats.oecd.org/RestSDMX/sdmx.ashx/GetKeyFamily/REFSERIES/OEC
D/?resolveRef=true">
<KeyFamilyRef>REFSERIES</KeyFamilyRef>
<Series>
<SeriesKey><Value value="FRA" concept="LOCATION"/><Value
value="YPTTTTL1_ST" concept="SUBJECT"/><Value value="A"
concept="FREQUENCY"/></SeriesKey>
<Attributes><Value value="P1Y" concept="TIME_FORMAT"/></Attributes>
<Obs><Time>2010</Time><ObsValue value="62927.11"/></Obs>
<Obs><Time>2011</Time><ObsValue value="63249.09"/></Obs>
</Series>
</DataSet></message:MessageGroup>
OECD.Stat - the OECD Data Warehouse
Solution - was the first dissemination system
to provide a means for disseminating data in
the internationally agreed standard, SDMX.
Having implemented a fully conforming 2.0
SDMX web service in 2007

7. Automated data exchange
Then began to introduce a number of
automated data exchanges through SDMX
2.0 interface with several organisations and
data-resellers / aggregators

8. Flexible
However although SDMX is highly flexible

9. but complex
It is also highly complex. This slowed its
uptake and also reuse

10. {“the oecd experience”:“2012-beyond”}
So entered a period when users needs
changed and with an ever increasing demand
for reuse.

11. KISS
Then started the thinking about KISS…

12. KISS Principle
No not the band but the KISS Principle: Keep
it simple, stupid (and short)!

13. Date: Mon, 12 Mar 2012 13:30:01 CET
Location: OECD, Paris
Meeting: OECD’s Statistical Information System
Collaboration Community (SIS-CC) workshop
Presenter: Xavier Badosa
Title: Standards for statistical data dissemination
Content-Type: Simplify+JSONify/SDMX
In March 2012 the 2nd OECD’s Statistical
Information System Collaboration Community
(SIS-CC) workshop took place with a
presentation by Xavier Badosa to push for a
simplified SDMX and create a common
JSON based format for statistical
dissemination.

14. XML
JSON
SDMX-JS?
SDMX-ML
Xavier proposed to take SDMX-ML (XML)
and together with JSON, a JavaScript Object
Notation (JSON) text-based open standard
designed for human-readable data
interchange.

15. "generic:Attributes" : {"generic:Value" : {"@concept" :
"OBS_STATUS", "@value" : "P"}}}, {"generic:Time" : "2011-
04", "generic:ObsValue" : {"@value" :
"106.56246"}, "generic:Attributes" : {"generic:Value" : {"@concept" :
"OBS_STATUS", "@value" : "P"}}}, {"generic:Time" : "2011-
05", "generic:ObsValue" : {"@value" :
"113.26596"}, "generic:Attributes" : {"generic:Value" : {"@concept" :
"OBS_STATUS", "@value" : "P"}}}, {"generic:Time" : "2011-
06", "generic:ObsValue" : {"@value" :
"114.22037"}, "generic:Attributes" : {"generic:Value" : {"@concept" :
"OBS_STATUS", "@value" : "P"}}}, {"generic:Time" : "2011-
07", "generic:ObsValue" : {"@value" :
"108.77534"}, "generic:Attributes" : {"generic:Value" : {"@concept" :
"OBS_STATUS", "@value" : "P"}}}, {"generic:Time" : "2011-
08", "generic:ObsValue" : {"@value" :
"116.37424"}, "generic:Attributes" : {"generic:Value" : {"@concept" :
"OBS_STATUS", "@value" : "P"}}}, {"generic:Time" : "2011-
09", "generic:ObsValue" : {"@value" :
“XMLish”
JSON
To create an XMLish JSON!

16. Standard definition led by SDMX TWG sub group
This was the start of an important parallel
evolution in SDMX led by a sub group of the
SDMXTWG.

17. {"sdmx-proto-json": "2013-03-26",
"structure": {
"dimensions": [
{ "id": "LOCATION", "name": "Country", "role": null,
"codes": [{"selected": true, "id": "FRA", "name":
"France"}]},
{ "id": "SUBJECT", "name": "Subject", "role": null,
"codes": [{"selected": true, "id":
"YPTTTTL1_ST", "name":
"Population mid-year estimates
Total, Annual, ('000)"}]},
{ "id": "FREQUENCY", "name": "Frequency", "role": null,
"codes": [{"selected": true, "id": "A", "name":
"Annual"}]},
{ "id": "TIME_PERIOD", "name": "Time", "role": null,
"codes": [{"selected": false, "id": "2010", "name":
"2010"},
{"selected": false, "id":
"2011","name": "2011"}]}
]},
"header": {"name": "Reference Series", "id": "none", "test":
"false",
"truncated": "false", "prepared": "2013-04-15T12:03:41",
"sender": {"id": "OECD", "name": "Organisation for Economic
Co-operation and Development"}},
"dataSets": []
}
{"sdmx-proto-json": "2013-02-05",
"structure": {"structure": "BISWEB_EERDATAFLOW", "href":
"todo",
"components": {
"LOCATION": {"id": "LOCATION", "name": "Country",
"values": [{"id": "FRA",
"name": "France"}]},
"SUBJECT": {"id": "SUBJECT", "name": "Subject",
"values": [{"id":
"YPTTTTL1_ST", "name":
"Population mid-year
estimates Total, Annual, ('000)"}]},
"FREQUENCY": {"id": "FREQUENCY", "name": "Frequency",
"values": [{"id": "A",
"name": "Annual"}]},
"TIME_PERIOD": {"id": "TIME_PERIOD", "name": "Time",
"values": [{"id":
"2010", "name": "2010"},
{"id": "2011", "name": "2011"}]},
"packaging": {
"dataSetDimensions": [],
"seriesDimensions": ["LOCATION", "SUBJECT",
"FREQUENCY"],
"observationDimensions": ["TIME_PERIOD"],
}}},
"header": {"name": "Reference Series", "id": "none", "test":
"false",
"truncated": "false", "prepared": "2013-04-15T13:48:40",
"sender": {"id": "OECD", "name": "Organisation for Economic
Co-operation and Development"}},
"dataSets": []
}
Two formats proposed
CodeIndex Slice
In late 2012 with 2 formats being
proposed, JSON-Codeindex and JSON-
Slice, the OECD set about creating a API to
service users with the 2 proposed JSON
based formats. The OECD SDMX-PROTO-
JSON API was launched in early 2013 ahead
of the 2013 SIS-CC workshop.

18. {"sdmx-proto-json": "2013-03-26",
"structure": {
"dimensions": [
{ "id": "LOCATION", "name": "Country", "role": null,
"codes": [{"selected": true, "id": "FRA", "name":
"France"}]},
{ "id": "SUBJECT", "name": "Subject", "role": null,
"codes": [{"selected": true, "id": "YPTTTTL1_ST",
"name":
"Population mid-year estimates
Total, Annual, ('000)"}]},
{ "id": "FREQUENCY", "name": "Frequency", "role": null,
"codes": [{"selected": true, "id": "A", "name":
"Annual"}]},
{ "id": "TIME_PERIOD", "name": "Time", "role": null,
"codes": [{"selected": false, "id": "2010", "name":
"2010"},
{"selected": false, "id":
"2011","name": "2011"}]}
]},
"header": {"name": "Reference Series", "id": "none", "test":
"false",
"truncated": "false", "prepared": "2013-04-15T12:03:41",
"sender": {"id": "OECD", "name": "Organisation for Economic
Co-operation and Development"}},
"dataSets": []
}
{"sdmx-proto-json": "2013-02-05",
"structure": {"structure": "BISWEB_EERDATAFLOW", "href":
"todo",
"components": {
"LOCATION": {"id": "LOCATION", "name": "Country",
"values": [{"id": "FRA",
"name": "France"}]},
"SUBJECT": {"id": "SUBJECT", "name": "Subject",
"values": [{"id":
"YPTTTTL1_ST", "name":
"Population mid-year
estimates Total, Annual, ('000)"}]},
"FREQUENCY": {"id": "FREQUENCY", "name": "Frequency",
"values": [{"id": "A",
"name": "Annual"}]},
"TIME_PERIOD": {"id": "TIME_PERIOD", "name": "Time",
"values": [{"id":
"2010", "name": "2010"},
{"id": "2011", "name": "2011"}]},
"packaging": {
"dataSetDimensions": [],
"seriesDimensions": ["LOCATION", "SUBJECT",
"FREQUENCY"],
"observationDimensions": ["TIME_PERIOD"],
}}},
"header": {"name": "Reference Series", "id": "none", "test":
"false",
"truncated": "false", "prepared": "2013-04-15T13:48:40",
"sender": {"id": "OECD", "name": "Organisation for Economic
Co-operation and Development"}},
"dataSets": []
}
Converged into…
The 2 formats, after much discussion and a
real world developer community
feedback, converged.

19. Simplified SDMX
{“SDMX”:“JSON”}
To create a single format…

20. {“SDMX”:“JSON”}
{"header":{"id":"ed9dd4bb-3725-4064-b173-842887bae108","test":false,"prepared":"2013-08-
23T23:08:34Z","sender":{"id":"OECD","name":"Organisation for Economic Co-operation and
Development"},"request":{"uri":"http://stats.oecd.org/SDMX-
JSON/data/MEI_FIN/IRLT.AUS+AUT+BEL+CAN+CHL+CZE+DNK+EST+FIN+FRA+DEU+GRC+HUN+ISL+IRL+ISR+ITA+JPN+KOR+LUX+MEX+NLD+NZL+NOR
+POL+PRT+SVK+SVN+ESP+SWE+CHE+TUR+GBR+USA+EA17+SDR+NMEC+BRA+CHN+IND+IDN+RUS+ZAF.M/OECD?startPeriod=2012&endPeriod=2013"
}},"dataSets":[{"action":"Informational","annotations":[],"attributes":[],"series":{"0:0:0":{"attributes":[0],"observa
tions":{"0":[3.795],"1":[3.97],"2":[4.15],"3":[3.8575],"4":[3.2775],"5":[2.995],"6":[2.8875],"7":[3.1875],"8":[3.0925]
,"9":[3.0225],"10":[3.0875],"11":[3.2275]}},"0:1:0":{"attributes":[0],"observations":{"0":[3.27],"1":[3],"2":[2.87],"3
":[2.83],"4":[2.49],"5":[2.29],"6":[2.07],"7":[1.97],"8":[2.04],"9":[2.02],"10":[1.85],"11":[1.77]}},"0:2:0":{"attribu
tes":[0],"observations":{"0":[4.01],"1":[3.48],"2":[3.33],"3":[3.42],"4":[3.22],"5":[3.09],"6":[2.74],"7":[2.55],"8":[
2.59],"9":[2.53],"10":[2.4],"11":[2.19]}},"0:3:0":{"attributes":[0],"observations":{"0":[3.9171],"1":[3.7524],"2":[3.2
886],"3":[3.3889],"4":[3.5291],"5":[3.4122],"6":[3.2504],"7":[3.0141],"8":[2.4256],"9":[2.3061],"10":[2.2492],"11":[2.
1039]}},"0:33:0":{"attributes":[0],"observations":{"0":[8.35],"1":[8.06],"2":[8.09],"3":[8.18],"4":[8.13],"5":[8.26],"
6":[8.09],"7":[8.17],"8":[8.25],"9":[8.21],"10":[7.96],"11":[8.1]}},"0:34:0":{"attributes":[0],"observations":{"0":[8.
4],"1":[8.23],"2":[8.37],"3":[8.28],"4":[8.28],"5":[8.16],"6":[7.52],"7":[7.48],"8":[7.4],"9":[7.67],"10":[7.64],"11":
[7.37]}}}}],"structure":{"uri":"http://stats.oecd.org/SDMX-
JSON/dataflow/MEI_FIN","name":"MEI_FIN","description":"Monthly Monetary and Financial Statistics
(MEI)","dimensions":{"dataSet":[],"series":[{"keyPosition":0,"id":"SUBJECT","name":"Subject","values":[{"id":"IRLT","n
ame":"Long-term interest rates, Per cent per
annum"}],"role":null},{"keyPosition":1,"id":"LOCATION","name":"Country","values":[{"id":"AUS","name":"Australia"},{"id
":"AUT","name":"Austria"},{"id":"BEL","name":"Belgium"},{"id":"CAN","name":"Canada"},{"id":"CHL","name":"Chile"},{"id"
:"CZE","name":"Czech
Republic"},{"id":"DNK","name":"Denmark"},{"id":"FIN","name":"Finland"},{"id":"FRA","name":"France"},{"id":"DEU","name"
:"Germany"},{"id":"GRC","name":"Greece"},{"id":"HUN","name":"Hungary"},{"id":"ISL","name":"Iceland"},{"id":"IRL","name
":"Ireland"},{"id":"ISR","name":"Israel"},{"id":"ITA","name":"Italy"},{"id":"JPN","name":"Japan"},{"id":"KOR","name":"
Korea"},{"id":"LUX","name":"Luxembourg"},{"id":"MEX","name":"Mexico"},{"id":"NLD","name":"Netherlands"},{"id":"NZL","n
ame":"New
Zealand"},{"id":"NOR","name":"Norway"},{"id":"POL","name":"Poland"},{"id":"PRT","name":"Portugal"},{"id":"SVK","name":
"Slovak
Republic"},{"id":"SVN","name":"Slovenia"},{"id":"ESP","name":"Spain"},{"id":"SWE","name":"Sweden"},{"id":"CHE","name":
"Switzerland"},{"id":"GBR","name":"United Kingdom"},{"id":"USA","name":"United States"},{"id":"EA17","name":"Euro area
(17 countries)"},{"id":"RUS","name":"Russian Federation"},{"id":"ZAF","name":"South
Africa"}],"role":null},{"keyPosition":2,"id":"FREQUENCY","name":"Frequency","values":[{"id":"M","name":"Monthly"}],"ro
le":null}],"observation":[{"keyPosition":0,"id":"TIME_PERIOD","name":"Time","values":[{"id":"2012-01","name":"Jan-
2012"},{"id":"2012-02","name":"Feb-2012"},{"id":"2012-03","name":"Mar-2012"},{"id":"2012-04","name":"Apr-
2012"},{"id":"2012-05","name":"May-2012"},{"id":"2012-06","name":"Jun-2012"},{"id":"2012-07","name":"Jul-
2012"},{"id":"2012-08","name":"Aug-2012"},{"id":"2012-09","name":"Sep-2012"},{"id":"2012-10","name":"Oct-
2012"},{"id":"2012-11","name":"Nov-2012"},{"id":"2012-12","name":"Dec-
2012"}],"role":"time"}]},"attributes":{"dataSet":[],"series":[{"id":"TIME_FORMAT","name":"Time
format","values":[{"id":"P1M","name":"Monthly"}],"role":null}],"observation":[]},"annotations":[]},"errors":[]}

21. BETA: stats.oecd.org/opendataapi/Json.htm
{“SDMX”:“JSON”}
<request> ::= "http://stats.oecd.org/SDMX-JSON/data/" <dataset name> "/" <filter
expression> "/" <agency name> [ "?" <additional parameters> ] <additional
parameters> ::= "startPeriod=" <start time> [ "&" <additional parameters> ] |
"endPeriod=" <end time> [ "&" <additional parameters> ] | "json-lang=" <language> [
"&" <additional parameters> ] | "json-format=" <json format> [ "&" <additional
parameters> ] | "dimensionAtObservation=" <code of dimension at observation level> [
"&" <additional parameters> ] | "AllDimensions" [ "&" <additional parameters> ]
<filter expression> ::= <dimensions> <dimensions> ::= <dimension> [ "." <dimensions>
] <dimension> ::= [ <dimension values> ] <dimension values> ::= <dimension value> [
"+" <dimension values> ] <language> ::= "en" | "fr"
The OECD has since updated the API to
reflect this single format that can be
accessed through the link shown.

22. Supporting the wider developer community
However, the OECD recognised the need to
reach much wider and thus.

23. Standard protocol for retrieving any kind of data over HTTP
Introduced an OData API, an implementation
of the Microsoft OData Protocol connected to
OECD.Stat.

24. Simple lightweight data format
{"odata.metadata":
"http://stats.oecd.org/OECDStatWCF_OData/OData.svc
/$metadata#DotStat.OData.DataSource.DatawarehouseE
ntities/REFSERIES",
"value": [
{ "FREQUENCY": "A",
"LOCATION": "FRA",
"SUBJECT": "YPTTTTL1_ST",
"TIME": "2010",
"Flags": "",
"Metadata": null,
"Value": 62927.11},
{ "FREQUENCY": "A",
"LOCATION": "FRA",
"SUBJECT": "YPTTTTL1_ST",
"TIME": "2011",
"Flags": "",
"Metadata": null,
"Value": 63249.09}
]
}
to retrieve data in a simple and lightweight
data format, both JSON and XML.

25. Reaching out to the wider developer community
Having built it, we needed them to come.

26. OpenDataAPI
The OECD’s Statistical Information System
Collaboration Community
Open Data Initiative for Developers
So in early 2013 ahead of the
OECD’s Statistical Information
System Collaboration Community
workshop in April 2013, the Open
Data Initiative for Developers was
launched.

27. OpenDataAPI
Objectives: benefits and ease of use
The Open Data Initiative for Developers objective
was to demonstrate the benefits and the ease of use of
the OECD.Stat Open Data web services by exposing
the OECD data through two new beta web services to
interested web developers to develop innovative
applications based on OECD public data, and to gather
feedback from developers to improve the web services
in future versions.

28. The Open Data Initiative for Developers was a
great success concluding at the SIS-CC 2013
Workshop held in Paris, when a group of
developers reviewed each submission, shortlisted
and presented the final submissions and overall
feedback to the wider group.

29. http://www.drasticdata.nl/ProjectOECD
http://www3.inegi.org.mx/rnm/sdmxhttp://www.uis.unesco.org/das/Country/OdataTest
Shortlisted submissions

30. SDMX-JSON: very simple to use
OData: simple model and is
popular with developers
Overall the feedback was
positive and directly
contributed to the further
development of the APIs

31. {“SDMX”:“JSON”}
…and in particular the final single JSON format

32. Potential future formats
As already mentioned the OECD may
introduce APIs serving more formats that could
include:
- Google Data (a REST-inspired technology),
- Google Dataset Publishing Language
(DPSL)
- Google KML, a Geospatial file format

33. Wider benefits
The APIs are also helping to contribute to the
OECD data portal project and supported the
development of a beta mobile version of
OECD.Stat.

34. References
1. sdmx.org
2. xavierbadosa.com/oecd
3. github.com/sdmx-twg
4. odata.org
5. stats.oecd.org/opendataapi/Json.htm (BETA)
6. stats.oecd.org/opendataapi/OData.html (BETA)

35. Jonathan Challener, OECD
jonathan.challener@oecd.org
@Challener
Thank you
{“Providing easier access to data”:“Through APIs”}
The OECD Delta Project
IMAODBC - Neuchatel, 16-20 September 2013