OpenRefine is a tool for working with messy data that allows for data profiling, cleaning, extension, and ETL prototyping. The document outlines two use cases - cleaning inconsistencies in a city thesaurus XML file with over 5,000 concepts, and preparing city data for a contest by cleaning duplicates, adding new data from another project, and using the tool's project history. OpenRefine is a cross-platform web application that can be used as an intermediate step between systems for data manipulation and cleaning.

1. 2013-06-26
Knowledge Workers Toronto
Methods
1
We are surrounded by data

2. 2013-06-26
Knowledge Workers Toronto
Methods
2
... by MESSY data
- Multiple standards and formats
Structured vs unstructured
Field nomination and format varies ...
- Human Error (misspellings, errors, etc)
- Non-normalized inputs (free-text entries)
- Incomplete data (laziness)
....

3. 2013-06-26
Knowledge Workers Toronto
Methods
3

4. 2013-06-26
Knowledge Workers Toronto
Methods
4
OpenRefine the
- Swiss army knife for data manipulation!
- glue step between your IT systems

5. 2013-06-26
Knowledge Workers Toronto
Methods
5
What's OpenRefine
(former Google Refine, former Gridworks)
- A Cross platform Web Application that runs
locally
- A Community based project hosted on GitHub
- Which have two distributions and multiple
extensions
- Something between a spreadsheet and SQL

6. 2013-06-26
Knowledge Workers Toronto
Methods
6
Four use cases
1. Data Profiling *
2. Data Cleaning *
3. Data extension *
4. ETL (Extract Transform Load) Prototyping

7. 2013-06-26
Knowledge Workers Toronto
Methods
7
File 1: Data Profiling &
Cleaning
City Subject Thesaurus XML file with 5431
concept.
What we will do:
- Explore the file
- Fix inconsistencies
- Transpose / Merge fields

8. 2013-06-26
Knowledge Workers Toronto
Methods
8
File 2: The Economist Best
City Contest 2012
Prepare Data for an application to the Economist
Intelligence Units (EIU) Best City Contest 2012
using G. Hofstede Values Survey Module 2008
What we will do:
- Clean Duplicate
- Create New Data
- Add data from a different project
- Use Project History

9. 2013-06-26
Knowledge Workers Toronto
Methods
9
OpenRefine
http://openrefine.org
@OpenRefine
Martin Magdinier
martin.magdinier@gmail.com
@magdmartin
Thanks!

10. 2013-06-26
Knowledge Workers Toronto
Methods
10
DESCRIPTOR The preferred term
FAC Facet
SC Subject category of the term
SN Scope note
SRC Source of term
UF Used for
BT Broader term
City Subject Thesaurus
Legend
NT Narrower term
RT Related term
STA Term status
INP Input date
APP Approval date
UPD Modified date
TNR Term number