Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Â
Easy Mechanical Turk Integration with Houdini
1. Houdini
On Demand Labor API
Easy Mechanical Turk Integration
with Houdini
Philly AWS December Meetup
Date: December 13, 2010
By: Chris Conley and Jameson Detweiler
12/13/2010
Tuesday, December 14, 2010
2. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Mechanical Turk
⢠Marketplace for work
⢠~ 500k workers
⢠Accessible via API
12/13/2010
Tuesday, December 14, 2010
3. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Task Examples
⢠Moderation - images, comments, reviews, video
⢠Categorization - products, best image, tagging
⢠Ads - moderation, keywords
⢠Audio Transcription / Video Captioning
⢠Cleanup - proofreading, contact reconciliation, review editing
⢠PDF/Image/Handwriting to Text
⢠Research - contact info, document search
12/13/2010
Tuesday, December 14, 2010
4. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Turk Overview
⢠Workers and Requesters
⢠HITs
⢠Questions
⢠Assignments
⢠QualiďŹcations
⢠NotiďŹcations
⢠Blocking Workers
12/13/2010
Tuesday, December 14, 2010
5. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Worker View
12/13/2010
Tuesday, December 14, 2010
6. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Creating HITs
WYSIWYG API
⢠Easy ⢠More Flexible
⢠Great for testing ⢠But Complicated
the waters
⢠Programmatic
⢠No integration Integration
12/13/2010
Tuesday, December 14, 2010
7. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
WYSIWYG
12/13/2010
Tuesday, December 14, 2010
8. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
WYSIWYG
12/13/2010
Tuesday, December 14, 2010
9. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
API
http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html
12/13/2010
Tuesday, December 14, 2010
10. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
HIT Questions
⢠XML
⢠External Question
12/13/2010
Tuesday, December 14, 2010
11. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Turk API Libraries
⢠ruby-aws http://rubyforge.org/projects/ruby-aws/
⢠rTurk https://github.com/mdp/rturk
⢠Turkee https://github.com/aantix/turkee
12/13/2010
Tuesday, December 14, 2010
12. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Questions about Turk?
12/13/2010
Tuesday, December 14, 2010
13. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Houdini
⢠Simple
⢠QualiďŹed results
⢠Automatic
⢠Trusted Workers
12/13/2010
Tuesday, December 14, 2010
14. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Houdini APIs
Generic Optimized
⢠Simple ⢠Image Moderation
⢠Exact ⢠Ad Moderation
⢠Multi-answer ⢠Data Reconciliation
⢠Data Extraction
⢠...
12/13/2010
Tuesday, December 14, 2010
15. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Simple API
⢠One answer per task
⢠Cheap
⢠Image Moderation, Low proďŹle content
12/13/2010
Tuesday, December 14, 2010
16. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Simple API Example
12/13/2010
Tuesday, December 14, 2010
17. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Exact API
⢠Multiple workers
⢠Ensures all answers match
⢠Returns the âďŹnal answerâ
⢠If not, escalated to trusted worker group
⢠More expensive than Simple
⢠Image Moderation, Data Harvesting, etc
12/13/2010
Tuesday, December 14, 2010
18. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Exact API Example
12/13/2010
Tuesday, December 14, 2010
19. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Multi-answer API
⢠Ask multiple turkers same question
⢠All answers are returned
⢠Surveys, feedback, answer qualiďŹcation
on your end
12/13/2010
Tuesday, December 14, 2010
20. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Image Moderation API
⢠No need to build form_html
⢠Just send the image url
⢠Optionally send custom instructions
and possible responses
12/13/2010
Tuesday, December 14, 2010
21. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Image Moderation Example
12/13/2010
Tuesday, December 14, 2010
22. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Image Moderation Demo
Source: https://github.com/chrisconley/cat_or_not
12/13/2010
Tuesday, December 14, 2010
23. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Admin Console
⢠Review Work
⢠Perform âTrustedâ tasks
⢠Worker Insights: Avg Completion Time,
# of workers
⢠Remove bad performing workers from
speciďŹc task groups
12/13/2010
Tuesday, December 14, 2010
24. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Getting Started
⢠Get an API Key - http://houdiniapi.com/sign-up/
⢠Tunnlr - http://tunnlr.com
⢠Sandbox - http://houdini-sandbox.heroku.com
12/13/2010
Tuesday, December 14, 2010
25. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Pricing Example
Monthly Fee Images Per Month
$20 200
$99 1500
12/13/2010
Tuesday, December 14, 2010
26. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Other Solutions
⢠CrowdďŹower
⢠Microtask
⢠CrowdCloud
12/13/2010
Tuesday, December 14, 2010
27. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Resources
Houdini Docs:
http://houdiniapi.com/documentation
Mturk Docs:
http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html
12/13/2010
Tuesday, December 14, 2010
28. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Thanks!
Chris Conley Jameson Detweiler
chris@houdiniapi.com jameson@houdiniapi.com
@chrisconley @jaymstr
12/13/2010
Tuesday, December 14, 2010