Presentation at the Centre for Investigative Journalism Summer School, July 2019

1. Paul Bradshaw
Leanpub.com/scrapingforjournalists*
Scraping
for stories

2. Why scraping
How to spot
opportunities for scraping
Tools and traits: what can
be scraped, and how
Why and how

3. Automating the repetitive
gathering of data, e.g.
Data from same page every
day (e.g. social media)
Data from multiple pages
Multiple documents:
spreadsheets, PDFs
What is scraping?

4. Why is a government
website carrying fake
jobs?
Aron Pilhofer, News Rewired

5. https://www.youtube.com/watch?v=Efr-VEkwWoM

6. http://www.mirror.co.uk/news/uk-news/singer-best-vocal-range-uk-4323076

8. http://doctors.ajc.com/

9. https://www.youtube.com/watch?v=DWRGqmywNYs&t=2m10s
https://nos.nl/op3/artikel/2248432-zo-dun-moet-je-echt-zijn-als-je-model-wil-worden.html
https://datajournalismawards.org/projects/this-is-how-thin-you-have-to-be-to-walk-the-catwalk/

11. You want to investigate
allegations of match
fixing, what data do you
scrape?

12. One scraper, many stories

15. Patterns, hunches

16. Anticipate published data
exemption
‘Second source’ for FOI
data
Combine with FOI

19. Gives you a long term
insight into the issue
Allows you to spot things
being removed and added
Scheduled scrapes

21. https://www.bbc.co.uk/news/health-48624554

22. New entries - or
disappearing ones

25. Data outside scope of FOI
Repetitive work, e.g. 1000s
of documents
Information that changes
regularly
Sources that follow a pattern
Stories that suit scraping

26. Data that can be obtained
through asking, FOI, API etc.
Multiple sites with different
CMS
Small amounts of data/
repetition
Stories that don’t.

27. What makes a website
suitable for scraping?

28. *

29. *
http://www.w4mpjobs.org/SearchJobs.aspx?search=alljobs

30. *
URL parameters
https://stores.sainsburys.co.uk/

31. *

32. *
https://tfl.gov.uk/corporate/safety-and-security/road-safety/safety-enforcement-cameras

33. *
https://tfl.gov.uk/corporate/safety-and-security/road-safety/safety-enforcement-cameras

34. HTML <table>
HTML tag(s)
JSON file
XML
TXT
CSV or XLS
PDF
PDF which needs OCR
Document challenges

35. 1 page, changes
>1 pages, ‘next’ links
pages linked from 1 index
>1 pages, URL pattern
Search results URL pattern
Search results, uses cookie
Search results, login needed
Hosting challenges

37. Patterns
Look for structure in a
webpage - how are
elements distinguished?
Think code and text

38. *
Chrome: right-click >
Inspect

39. *
Inspector: right-click >
Copy…

43. *
A tip about URLs

44. This bit isn’t needed.

45. This bit is just for SEO.

46. You can put anything
there. Literally.

47. Do it now:
Identify an online
source of information
you might scrape
Think beyond tables:
what about series of
pages? Documents?

48. Paul Bradshaw
Leanpub.com/scrapingforjournalists*
Thank you.