Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Python Tutorial-Mining imgur images

1.997 Aufrufe

Veröffentlicht am

This tutorial is created for social scientists interested in grabbing data from the image-hosting site, Imgur (imgur.com).

Veröffentlicht in: Bildung

Python Tutorial-Mining imgur images

  1. 1. Curiosity Bits - curiositybits.com • This tutorial is created for social scientists interested in grabbing data from the image-hosting site, Imgur (imgur.com). • Find out more about Python for mining the social web, please visit Curiosity Bits (curiositybits.com). • Social-Metrics.org also hosts a series of Python tutorials on aggregating and analyzingTwitter/Facebook data. More at social-metrics.org CURIOSITY BITS© Get Imgur Data through Python
  2. 2. Curiosity Bits - curiositybits.com This tutorial shows you how to download images and the images’ meta- data (i.e., image title, description, source, upload time, etc.). CURIOSITY BITS© Imgur.com is a popular site for sharing selfies and images intended for persuasion.
  3. 3. Curiosity Bits - curiositybits.com CURIOSITY BITS© Images will be saved in a designated folder. The metadata will be saved in a SQLite database
  4. 4. Curiosity Bits - curiositybits.com CURIOSITY BITS© Imgur images are available through keyword search, reddit timelines and public albums. Here are the examples: • http://imgur.com/search?q=hillary images from the search using the keyword “hillary” • http://imgur.com/r/transtimelines images from a public reddit timeline called transtimelines • http://imgur.com/gallery/ROYAZ images from a public album called ROYAZ Three sources of Imgur images
  5. 5. Curiosity Bits - curiositybits.com Final note before we start: for simplicity, I am laying out only the most essential steps. Previous tutorials provide details about how to set up a Python programing environment, please visit curiositybits.com and click the PYTHON tab. CURIOSITY BITS©
  6. 6. Curiosity Bits - curiositybits.com 1. Install Anaconda Python (with Spyder and Ipython Notebook) 2. Install SQLite Browser 3. Install four essential Python packages (imgurpython, sqlalchemy, urllib, sqlite3) 4. Register a Imgur client to get client ID and client secret 5. Create a SQLite database for images from keyword search 6. Download images from the keyword search 7. Create a SQLite database for images from a reddit timeline 8. Download images from a reddit timeline 9. Create a SQLite database for images from a public album 10. Download images from a public album CURIOSITY BITS© Steps
  7. 7. Curiosity Bits - curiositybits.com CURIOSITY BITS© Install Anaconda Python • store.continuum.io/cshop/anaconda/ You can run Python codes in Spyder, which is a component in Anaconda Python Or use IPython Notebook, a web- based interactive Python environment
  8. 8. CURIOSITY BITS© This is what Spyder looks like.The left side of the window displays codes.The one at the bottom right shows you the results. IPython Notebook runs your codes on the web and displays results in your browser. Install Anaconda Python
  9. 9. Curiosity Bits - curiositybits.com CURIOSITY BITS© http://sqlitebrowser.org Install SQLite Browser
  10. 10. Curiosity Bits - curiositybits.com CURIOSITY BITS© • In the Command Prompt, use the command line pip install, followed by the package name, to install necessary packages. Required packages • Imgurpython • Sqlalchemy • Urllib • sqlite3 Install four essential Python packages
  11. 11. Curiosity Bits - curiositybits.com CURIOSITY BITS© type the following command lines in the command prompt: pip install imgurpython pip install sqlalchemy pip install urllib pip install sqlite3 Install four essential Python packages
  12. 12. Curiosity Bits - curiositybits.com Imgur FetcherV1 is a collection of codes used in this tutorial.You can download them at https://github.com/cosmopolitan van/imgur_curiositybits_v1 On the site, you can also find three examples of SQLite database.The database files end with the extention .sqlite CURIOSITY BITS© The Python codes used in the tutorial
  13. 13. Curiosity Bits - curiositybits.com CURIOSITY BITS© Sign in your Imgur account, go to https://api.imgur.com/oauth 2/addclient and get your “security clearance”. Register a Imgur client
  14. 14. Curiosity Bits - curiositybits.com CURIOSITY BITS© You need Client ID and Client secret to grab data from Imgur API Register a Imgur client
  15. 15. Curiosity Bits - curiositybits.com CURIOSITY BITS© Let’s get images with the keyword “hilliary” http://imgur.com/search?q=hillary We will download all available images, but at first, let’s get their metadata. By metadata, I mean, attributes related to each image. Examples of the metadata include image title, image description, image upload date, image link (which is what we will use to download images). Create a SQLite database for images from keyword search
  16. 16. You can right-click the link and download the file with the extension .py. Open the file in Spyder. Or, copy the entire block of codes into Ipython Notebook. Curiosity Bits - curiositybits.com CURIOSITY BITS© Use the code named Imgur search v1.py
  17. 17. CURIOSITY BITS© You don’t have to change anything in this block of codes. It is used for importing necessary Python packages.Think of packages as apps running on iOS. Open Imgur search v1.py in Spyder
  18. 18. Curiosity Bits - curiositybits.com CURIOSITY BITS© This is where you enter the keyword(s) you want to apply to the search.You can have multiple keywords, wrapped in parenthesis, and separated by a comma.
  19. 19. Curiosity Bits - curiositybits.com CURIOSITY BITS© This is where you enter the client ID and client secret generated in the previous step.
  20. 20. Curiosity Bits - curiositybits.com CURIOSITY BITS© No tweaking is needed for this block. But it gives you a sense of what data are to be collected.
  21. 21. Curiosity Bits - curiositybits.com CURIOSITY BITS© Go to line 136, this is where you specify the name of the SQLite database to be saved. If no absolute file path is given, the database will be saved in the same folder with your Python script (Imgur search v1.py).
  22. 22. Curiosity Bits - curiositybits.com CURIOSITY BITS© Imgur images are indexed into multiple pages. Here, 5 means that we are to get five pages of images from the keyword search.You can put 3 or 1 or 2, just play around to see what number gives you the most adequate, while at the same time manageable amount of data.
  23. 23. Curiosity Bits - curiositybits.com CURIOSITY BITS© Now, the tweaking is done! From the menu, chose Run-Configure and hit RUN
  24. 24. Curiosity Bits - curiositybits.com CURIOSITY BITS© This is the SQLite database that contains all the metadata.
  25. 25. Curiosity Bits - curiositybits.com CURIOSITY BITS© Notice that the last column is called multiple_images. Most image links will end with an extension name of a picture (.jpg, .png, .,bmp, .gif). If that is the case, the value on the column will be NO. But, some links contain multiple images, and you will see “this link contains multiple images…” warning in the column.
  26. 26. Curiosity Bits - curiositybits.com CURIOSITY BITS© Now, we have a saved SQLite database with image links.The next step is to use Python to download images from those image links Use the .py file named Imgur search_downloader v1.py Download images from the keyword search
  27. 27. Curiosity Bits - curiositybits.com CURIOSITY BITS© Go to line 72, and make sure the filename there matches the SQLite database we have just created. Download images from the keyword search
  28. 28. Curiosity Bits - curiositybits.com CURIOSITY BITS© Please create a new folder and enter the folder path here. Image filename is comprised of the image’s unique identifier id from Imgur API. Download images from the keyword search
  29. 29. Curiosity Bits - curiositybits.com CURIOSITY BITS© Again, this script downloads images from links with the extension of .jpg, .png, .gif, .bmp If an image link ends with anything but an image extension, the script will prompt a reminder message. Download images from the keyword search
  30. 30. Curiosity Bits - curiositybits.com CURIOSITY BITS© Now, execute the code and you will find the downloaded images in the folder you have just created. Download images from the keyword search
  31. 31. Curiosity Bits - curiositybits.com CURIOSITY BITS© Getting image from a reddit timeline is very similar to getting images from keyword search.This time, we will try executing the Python code in Ipython Notebook. Use the script called Imgur reddit timeline v1.py Create a SQLite database for images from a reddit timeline
  32. 32. Curiosity Bits - curiositybits.com CURIOSITY BITS© Create a new notebook in IPython Notebook and copy and paste the code from Imgur reddit timeline v1.py Create a SQLite database for images from a reddit timeline
  33. 33. Curiosity Bits - curiositybits.com CURIOSITY BITS© Enter the name of a reddit timeline. In the URL term, a reddit timeline displayed as http://imgur.com/r/buffalo Enter your client ID and client secret.
  34. 34. Curiosity Bits - curiositybits.com CURIOSITY BITS© Enter or change the filename of the SQLite database to be saved
  35. 35. Curiosity Bits - curiositybits.com CURIOSITY BITS© The number of pages to be grabbed…
  36. 36. Curiosity Bits - curiositybits.com CURIOSITY BITS© Click Run Cell The output will be saved in your IPython Notebooks folder. By default, it is …DocumentsIPython Notebooks
  37. 37. Curiosity Bits - curiositybits.com CURIOSITY BITS© Download images from a reddit timeline Use the .py file named Imgur reddit_downloader v1.py
  38. 38. Curiosity Bits - curiositybits.com CURIOSITY BITS© Download images from a reddit timeline Load the code in IPython Notebooks, Like what you do to download images from keyword search, make sure the file path in the code matches the database you have just created.
  39. 39. Curiosity Bits - curiositybits.com CURIOSITY BITS© Download images from a reddit timeline Specify the folder path, and run the code!
  40. 40. Curiosity Bits - curiositybits.com CURIOSITY BITS© Create a SQLite database for images from a public album Exactly as what we do in previous steps.There are only a few places in the script that need tweaking.You need to enter client ID, client secret, the name of the album, the file path of the database, the number of pages to be grabbed.Then you are all good to go! To create a SQLite database for images from a public album, use the script named Imgur album v1.py
  41. 41. Curiosity Bits - curiositybits.com CURIOSITY BITS© Download images from a public album To download images from a public album, use the script named Imgur alubm_downloader v1.py
  42. 42. Curiosity Bits - curiositybits.com CURIOSITY BITS© Having questions? Contact @cosmopolitanvan onTwitter

×