Slides from a presentation at the Centre for Investigative Journalism Summer School 2018

1. Paul Bradshaw
Leanpub.com/scrapingforjournalists*
Scraping
for stories

2. Why scraping
How to spot
opportunities for scraping
Tools and traits: what can
be scraped, and how
Why and how

3. Automating the repetitive
gathering of data, e.g.
Data from same page every
day (e.g. social media)
Data from multiple pages
Multiple documents:
spreadsheets, PDFs
What is scraping?

4. Why is a government
website carrying fake
jobs?
Aron Pilhofer, News Rewired

5. https://www.youtube.com/watch?v=Efr-VEkwWoM

6. http://www.mirror.co.uk/news/uk-news/singer-best-vocal-range-uk-4323076

10. http://blogs.ft.com/ftdata/2014/06/11/interactive-explore-the-statistical-identity-of-every-team-at-the-world-cup/?

11. http://www.nature.com/news/scientific-publishing-the-inside-track-1.15424

14. One scraper, many stories

16. http://www.private-eye.co.uk/registry

18. Patterns, hunches

19. http://helpmeinvestigate.com/olympics/olympic-torch-relay-youth-target-missed-by-over-1000-places/

20. Anticipate published data
exemption
‘Second source’ for FOI
data
Combine with FOI

23. Gives you a long term
insight into the issue
Allows you to spot things
being removed and added
Scheduled scrapes

26. New entries - or
disappearing ones

29. What makes a website
suitable for scraping?

30. *

31. *
http://www.w4mpjobs.org/SearchJobs.aspx?search=alljobs

32. *
URL parameters
https://stores.sainsburys.co.uk/

33. HTML <table>
HTML tag(s)
JSON file
XML
TXT
CSV or XLS
PDF
PDF which needs OCR
Document challenges

34. 1 page, changes
>1 pages, ‘next’ links
pages linked from 1 index
>1 pages, URL pattern
Search results URL pattern
Search results, uses cookie
Search results, login needed
Hosting challenges

36. Patterns
Look for structure in a
webpage - how are
elements distinguished?
Think code and text

37. *
Chrome: right-click >
Inspect

38. *
Inspector: right-click >
Copy…

42. *
A tip about URLs

43. This bit isn’t needed.

44. This bit is just for SEO.

45. You can put anything
there. Literally.

46. Do it now:
Identify an online
source of information
you might scrape
Think beyond tables:
what about series of
pages? Documents?

47. https://onlinejournalismblog.com/2013/09/18/ethics-in-data-journalism-mass-data-gathering-scraping-foi-and-deception/

48. Robots.txt
http://www.tcij.org/robots.txt

49. Database rights
Data copyright
Terms & conditions
Legal considerations

51. https://moveplanner.zoopla.co.uk/terms-and-conditions

52. Treat like any source:
build in TGTBT checks
Seek second sources
Seek right of reply/
confirmation
Data is just a lead

53. http://www.storybench.org/to-scrape-or-not-to-scrape-the-technical-and-ethical-challenges-of-collecting-data-off-the-web/

54. https://www.mediawiki.org/wiki/API:Main_page
Does it have an API?

55. https://github.com/BBC-Data-Unit/music-festivals

56. Paul Bradshaw
Leanpub.com/scrapingforjournalists*
Thank you.