VisibleGovernment.ca Expense Visualizer Pilot -- as presented at Montreal Ruby on Rails

1. Citizen Activism using Scrubyt and RoR

3.  Only partially available
online
 Formatted as web page or
PDF
 Hard to search
 Can’t subscribe
 Can’t visualize
 Can’t re-use

4. Publishing Structured Data Visualization
Feeds • Makes it easy to find new
• Ability to subscribe to patterns.
interesting data
• Data streams can be ‘mashed’
in new ways.
Collaborative Crowdsourcing
Organization • Combines skills and input of
• Tagging, Voting, Sharing large numbers of people

5. • Governments publish
Governments data streams
publish data
streams
• 3rd parties create tools for
analysis and oversight
3rd
Issues are
Party
Citizens
monitor data
• Citizens collaboratively
resolved
Tools streams monitor their
government
• Citizens detect issues,
Issues are
detected
give feedback
• Issues are resolved

7.  Government has little
incentive
▪ Usually has disincentive
Why can’t the
 Don’t want a single
government do monolithic solution
everything? ▪ Want to allow evolution of best-
of-breed tools
 Tools created by citizens, for
citizens

8.  Focus:
 US Congress
 California
Legislature
 Gives grants to
online
transparency
tools
 $3.5 M Seed

9. A recent US
Congress bill
Groups for Groups
bill against bill

10. Votes
Donations

11. Publishing Structured Data Visualization
Feeds • MAPLight makes relationship
• MAPLight is a mashup of between money and votes
data streams from different visible.
sources.
Collaborative Crowdsourcing
Organization • Thousands of journalists,
• Advocacy group tags advocates, and citizens can
donating companies as browse data and flag issues.
belonging to interest groups.

14.  Accelerate online transparency
Ideas  Raise Awareness
 With public
 With government
Skills  Raise Money
 Fund External Development:
Funds  Grants
 Contests

15. Prove
Concept
Get Publicity Direct Attention and
Money and to Online
Tools For Transparency
Raise
Awareness
Show What’s
Possible

17.  2003 Directive: Must
publish travel and
hospitality expenses
on the web
 No standards for
presentation defined

18. 124 Departments
- All different

23. Standardize Stream Visualize
• Scrape data into • Publish RSS • Provide basic
standard format feeds visualization app
• Run contest

25. 1. LEARNING TEMPLATE 2. PRODUCTION SCRAPER
Input Input
• Example Page • Any Page with
• Example Text Same Format
Output: Output:
• XML
• Production Scraper
• XML

28.  Create a system
where non-coders
can train a scraper.

29. PRO CON
 Ability to use ‘learning’  Learning mode fails hard
example (sometimes)
 Doesn’t always learn
 Syntax integrates XML
builder
 Supports all hpricot Xpath
operations
Note: For compatibility reasons, this project uses an older version of scrubyt.
Issues may be fixed in newer version.

30.  Create a system
where non-coders
can train a scraper.
.... Didn’t work.

31. Still need coders w/ the following expertise:
1. XPath XML resolution
2. Regular Expressions
3. Firebug

33. 1. Open This Link
2. Paste This Text

35. ...created in the
background

36. Go To Next Level

37. Split Level: Two Types of Links
Open This Link

38. Select Element
Get the XPath

39. Split Level: Two Types of Links

41. ...created in the
background

42. Test Random Reports
Send Home

43.  Goal: Finish scraping in one day
 12/124 Completed: 112 to go
 5-20 Volunteers
 5-20 min. per department
 Downloadable app w/ setup instructions
 Integrated examples
 Benefits:
 Excuse to use scrubyt, firebug
 On-site tutorial + guidance
 Easy intro to a Rails App

44. Jennifer Bell
visiblegovernment.ca