Slides used by David Opoku, 2015 School of Data fellow, for his skillshare about Scraping.

1. Intro to Data
ScrapingPRESENTED BY
DAVID SELASSIE OPOKU
@sdopoku
13 July 2015

2. Outline
1. Target audience
2. What is and Why Data Scraping?
3. Use cases
4. Basic steps & Best practices
5. Tools
6. Reference Resources

3. Target
Audience

4. This should be useful to ...
● Non-tech-savvy data journalists
● Advanced data journalists
● Web developers & data publishers
● School of Data fellows
● Open Data enthusiasts

5. What is &
Why Data
Scraping ?

6. Data Scraping: what is it ?
scrape [ verb ˈskrāp ]
: to remove from a surface by usually repeated strokes of an edged instrument
: to collect by or as if by scraping —often used with up or together <scrape up the
price of a ticket>
- Merriam Webster
“The transformation of unstructured data on the web, typically in HTML format, into
structured data that can be stored and analyzed in a central local database or
spreadsheet.”
- Wikipedia (web scraping)

7. When should you scrape data ?
● PDF Data
● HTML data
Machine-readable data

8. Example
Use Cases

9. Cases when you can scrape
● Create a dataset for a data workshop
● Create a database for a data -driven app
● Create a data visualisation for a story

10. Best
Practices

11. Best Practices For Scrapers
1. Scraping is not scary!
a. Use existing tools
2. Use a modern and friendly browser
a. Chrome, Firefox, Opera, Safari
b. Avoid Internet Explorer
3. Map out the process
a. Where does scraping fit in?

12. Best Practices For Data Publishers
1. Have a consistent structure
a. Websites
b. PDFs
2. Always think about your data end users
a. Before, during & after publishing

13. Steps
1. Map out the process/pipeline for your data project
2. Identify your data source (website, PDF, API?)
3. Decide on storage format for your scraped data
a. CSV file, Spreadsheet, Google docs
b. Database
4. Select scraping tool
5. Verify and Clean data

14. Tools

15. Tools: Web Browsers

16. Tools: Scraping Apps
1. Point and click
a. Scraper Google Chrome extension
b. ScraperWiki (Classic version)
c. Import.io, Kimono Labs, Webscraper.io
d. Tabula (PDF)
2. Programming (Python libraries)
a. Beautiful Soup
b. Pattern (PDF and HTML)
c. Scrapy

17. Tools: Storage & Sharing
1. Google Spreadsheets
2. Github
3. Datahub.io

18. Resources - Readings and Tools
1. Five data scraping tools for would-be data journalists
2. Making data on the web useful: scraping
3. Liberating HTML Data Tables
4. BeautifulSoup
5. Pattern
6. Scrapy
7. Datahub
8. Import.io
9. Kimono
10. Webscraper.io
11. Tabula