Presentation to the 8th Annual Public Sector ICT Conference. Irish Software Association. Tim Willoughby - Open Data, Bug Data Big Costs

1. Tim Willoughby
Big Data Open Data Big Costs?
ISA
24th October

2. Gartner Top 10 for 2012 -
• Released today by
Forbes

3. What has Cloud ever done for us?
Apart from Scale, Speed, Agility, Low Cost,
Enterprise Adoption, Enterprise Mapping, Open
Data, Standards, Google, API’s, etc
Open Street Maps, Map Servers,
GIS - More than just Location, Spatial Analysis and wider adoption now possible
3

4. How Much Data do you create?

5. How much data to you create?

6. Retailers..
• Assign each shopper a unique code - keeps tabs
on everything they buy.
– If you use a credit card or a coupon,
– or fill out a survey,
– or mail in a refund,
– or call the customer help line,
– or open an e-mail we’ve sent you
– or visit our Web site,
– we’ll record it and link it to your ID,”
• “We want to know everything we can.”
• “Cool is one click away from Creepy” (Fjord)

7. Your Demographic They Have or can get
• You
o your age,
o whether you are married, or got divorced
o and have kids, Your ethnicity,
• Where you live , the year you bought (or lost) your house, whether you’ve
moved recently,
• The number of cars you own.
• Distance to nearest Outlet,
• Worth
• Your estimated salary, if you’ve ever declared bankruptcy, Charity Giving
• What credit cards you carry in your wallet
• Education, where you went to college,
• Career - job history,
• Online -what Web sites you visit., the magazines you read, reading habits,
• The topics you talk about online,
• Likes- certain brands of coffee, cereal etc, political leanings,

8. Big Data goes with…
More Devices
More Access
More Apps
More Data…

11. 4 Global Themes
Social
DATA
Mobile Cloud

13. Radical Change…
Social has overtaken mobile ….
Mobile Browsing has overtaken desktop…

14. Change…
With or Without the Owners / Shareholders

16. So far ICT has not fundamentally
changed government
• 1990s: lCT expected
to make government
more
transparent, efficient
and user oriented
• 2005+: disillusion as
bureaucracy still in
existence
• Can Mobile / Data
/Cloud Help?
Jane E. Fountain – Gov 1.0 – Just Replicating the Silos on the Internet

17. Goverment dont always Understand
What the people Want?

18. What governments often Deliver

19. Open Government?
“If people don’t know what you’re doing, they
don’t know what you’re doing wrong”
“Jim Hacker”
Open Government (I980)

20. The World has changed…

23. Every day - you come across
multiple…
• - Devices
• - applications
• - access points
• - Networks
• - Locations
• - Servers
• - Data types
• - Data caches
• - Service providers
• - Operating systems
• - Storage locations
• - Encryption / security protocols..

24. My Smart Device is…. Very smart
Source : Peter Cochrane

25. It knows more
about me than
my partner…
Source : Peter Cochrane

27. Security is changing
• Security has to be appropriate
• Security has to be measured
• Can have things so secure that they are
unusable.

28. Citizen Expectation Changed?
Social
Media

29. The More data we have… the less we
(have time to) Understand it

32. Government as a Platform?

33. Mobile Development Spend is rapidly
overtaking PC Development Spend

34. BIG DATA?
 Massive network of services: water, sewage, drains …
 Need to know asset location for planning and maintenance
 Many databases, varying accuracy and provenance
 Context
 Ongoing street openings p.a.
 Safety!

35. Typical Individual effort
many hours, one map

36. OpenStreetMap, 2012
200,000 contributors, one map

39. Our Lives are Different…
• We communicate on — Facebook, Chat, Twitter,
MMS
• We Research Information — Blogs, eNews, Wiki,
YouTube
• We Buy — eBay, Amazon, Dell, Deal Done, etc…
• We Travel - Tripadvisor, Flickr, RyanAir, Hotels,
• We Meet / Retate - Linkedln, FaceBook, Friends
• We Play – xBox, Playstation, Online games, ¡TV
• We expect the Government to fit into these
paradigms too…

42. What is Big Data?
• Size beyond the ability of
commonly used software
tools
• Data growing challenges
• volume increasing
• velocity (speed of i/o)
• variety (data type/source)
• Examples:
computer tracking of
shipments, sales, suppliers, cust
omers, email, web traffic, social
network
From McKinsey Report

43. Why Now?. What is the Catalyst?
• People
– Web sites with 300+ million unique visitors/month
• Facebook
• Google
• YouTube
• Access
– 1.2 Billion active mobile broadband subscriptions
• Commodity (community) Hardware
– possibility for cost effective processing

44. Facts!
• “Data is a vital raw material of the information
economy, much as coal and iron ore were in the
Industrial Revolution.”
• “Mining and analyzing these big new data sets can
open the door to a new wave of
innovation, accelerating productivity and economic
growth.”
• We can exploit Internet-scale data sets to uncover
new businesses and predict consumer behavior and
location data
market shifts. sales business data social network data

45. Think…
• NY Traffic
– They have mountains of data
– More Daily
– Don’t Understand it, can’t use it
– What is the True Cost of Data Mining
– Systems, Software, Computers, Analysts, STORAGE
– Are we collecting, collating, etc the Right Data
– People being removed daily, replaced by machines
– Need to Stand Back….

46. How to deal with Lots of data
• 7TB a Day for Twitter…
• 10,000 CD’s
• 5m Floppy
• 225GB during this talk…
• How do you Store that…
• HD Write Speed- 80mb/s…
• 24.3 Hrs to write 7TB….
• Government Thinking is not in this space.. Yet..

48. Irish Government
• Decision – “All Government Invoices Published”
– Election Manifesto
– Election Promises
– Programme for Government
• Action
– None…
– Fear, Bad Reactions, Data Quality,

54. Stakeholders
• Private citizens
• Activists
• Coders
• Entrepreneurs
• NGO’s
• Academics
• Civil Servants
• Gov Institutions

55. Which Apps?
• Government – Do not try to second guess what
are useful applications!
• No one can predict what data to make open
based on potential Apps.
• There are a huge number of apps available for
smartphones. Impossible to guess a few years
back.
• Put Data out there. Data as a platform for others
to build on, as infrastructure.
• Apps you will like and hate, but it’s not up to you.

56. Applications?

57. Where is Open Data delivering?
• Participation / empowerment
• Transparency, democratic control
• New and better Government products and
services
• Impact measurement
• Efficient and Effective Government
• Data Journalism (Guardian Services)
• New knowledge / real innovation

58. The Internet of Things
Cars, Planes, Cameras,
Roads, Pipes, Public Cameras, Traffic Lights, Car
Parks, Pharma and Farms – over 1 Trillion

59. Smart Local Government?

60. Mass increase in new net devices
• Move to digital controls Wireless
integration
• Network standards
– TCP/IP
communications
– Wireless
communications
60

61. Or Wizards
GIS is now
Mainstream

62. Water Meters
• Read the Meter
• View or Pay the Bill
• Compare the Usage to the
Local or National
Averages
• Look for areas of Savings

63. Roadworks
• Link to System – Online
Road Works Control
(OLRWC)
• License for Road
Opening
• Cost of Reinstatement
(Local Authority Usage)
• License for any Street
usage.

64. Planning Applications
• View Applications
• Get notified of Changes
• Make Comments
• Make a Submission
• Notify Local Authority
of Issues?

65. Sports
• What is available now
• How do I get there
• What if I have to Cancel
• Pay for it now

66. Parking
• Select County
• Select Zone
• Select Price willing to
pay?
• Where is the Cheapest

67. open source / open data / cloud
the fix your street model

68. The Reality of
FixYourStreet

69. But all of these are static…..
But… would you
cross the road
based on 5 min
old information

70. Why Open Data
• Because the Data already belongs to everyone!
• Because there is Law, Inspire Directive - PSI
• It helps change the incentives, from Fail Big, Fail Slow to
Fail Small, Fail Fast, and Fail Forward (Innovation – More
eyes)
• Open Data will allow issues/problems to be seen, making
them less likely to be repeated (learning from other people’s
failures)
• Open Data use can allow Incremental Innovation: - stand on
the shoulders of giants. Don’t reinvent the wheel — use or
improve someone else’s
• Open Data quality can be increased over time — use and
reuse by other people and combining with other data
identifies (and helps solve) quality issues far more quickly
and cheaply than internal checking

71. How do we measure Success?
• Usage and Apps
• Critical Mass of Companies
• Search Results (Bing and Google Interested)
• Business Committed to OD
• Laws (Prison for breaking them)
• If Open Data doesn’t cause difficulty for the
Publisher to seek excellence then it isn’t
working

72. Where next - Augmented Reality
Bring together Big Data, Visualisation, GIS, GPS -
Bringing Cloud, Social, Data and Mobile
together…

73. What is Augmented Reality?

76. Water Meter

77. Water Meter

79. Understand what are the Driving
Forces…
• Government Focus
– Understanding, Efficiency, Accountability
• Technology Innovation Focus
– Data as a Platform, Semantic Web
• Reward Focus
– Profit, Recognition
• Digitising Government Focus
– Computerisation / Technology Drive
• Problem Solving Focus
– New Skills needed to work on new Challenge
• Social / Public Sector / Enterprise
– More Focussed Services

80. Conclusions
• Change is constant
• Cloud is the Future
• Social is a Reality
• Mobile is already taking over
• Data has to keep up…

81. References
• We live in “Flat Land”
– There is danger in
making
representations more
seductive than the
truth
• Envisioning
Information - Tufte