SlideShare a Scribd company logo
1 of 23
Download to read offline
KissKissBan
A Competitive Human Computation
   Game for Image Annotation
Chien-Ju Ho*, Tao-Hsuan Chang+, Jong-Chuan Lee+,
        Jane Yun-jen Hsu+, Kuan-Ta Chen*
                 *Academia Sinica
           +National Taiwan University
Human Computation Game



Collaborative Mechanisms
Coalition Problem

Limited Data Diversity
The ESP Game

     • Coalition problem
       • partner with themselves
       • unified strategy
     • Limited diversity
       • easy words first
Competitive Mechanism?
Player 1   Player 2




  DOG      CUTE
FLOWER     WOOD
 FLOOR      DOG
Player 1       Player 2




          DOG          CUTE
        FLOWER         WOOD
         FLOOR          DOG

DELIVER EMOTION BY TYPING THE SAME WORD
Couple         Couple




          DOG          CUTE
        FLOWER         WOOD
         FLOOR          DOG

DELIVER EMOTION BY TYPING THE SAME WORD
Blocker
Couple             Couple




         DOG
  DOG              CUTE
         WHITE
FLOWER             WOOD
 FLOOR              DOG
Blocker
Couple                             Couple




                 DOG
  DOG                             CUTE
                 WHITE
FLOWER                            WOOD
 FLOOR                             DOG

            KissKissBan
     A Competitive Human Computation
        Game for Image Annotation
Blocker
  Couple                                    Couple




                       DOG                  CUTE
    DOG Blocked
                       WHITE                WOOD
  FLOWER
                                           FLOWER
   FLOOR      The blocked words
-5s game time                           -5s game time

    1. First, the blocker has 7 seconds to type words
    2. Then, the couples has 30 seconds to type words
Blocker
Couple                                   Couple




                    DOG                  CUTE
  DOG
                    WHITE                WOOD
FLOWER
                                        FLOWER
 FLOOR
   Couples win for matching within time limit
 1. First, the blocker has 7 seconds to type words
 2. Then, the couples has 30 seconds to type words
Screenshots of KissKissBan
Intuitions Behind the Game
Incentive Structure

• Zero sum game
 • strict competitive
 • players can not benefit from coalition
• Different decision time periods
 • harder words, longer consideration [Barry et al. 2001]


 • diverse data contribution
Small-scale Experiment

• Image Source
 • 125 images randomly from ESP dataset
• Experiment Setup
 • 17-player gameplay survey
 • Data collection on Mechanical Turk
Gameplay Survey


• Fun
 • 3.76/5 overall rating
 • 15/17 players claim that would play again
 • over 60% like to play as a blocker
MTurk Experiment

• Collecting data from anonymous users
• Publish two games for comparison
 • KissKissBan
 • ESP Game without taboo words
• Each game is played for ~5000 rounds
Example of Collected Data
                   ESP    ML-KKB     BL-KKB
                          beach 3
                                       sea 9
                          water 3
                  man 21              man 8
                           sand 3
                 beach 10            ocean 3
                            sea 2
                 karate 5            black 1
                           ninja 1
                 water 1             china 1
                          kungfu 1
                                      sand 1
                          ocean 1

ML-KKB: matching between couples
BL-KKB: matching between couple and blocker
MTurk Experiment

• Data property
 • For each image
   • ESP: 39.96 labels, 6.56 distinct labels
   • KKB: 44.17 labels, 11.53 distinct labels
• A more diverse player contribution
Conclusion

• We proposed a competitive human
  computation game, KissKissBan.
• KissKissBan has two nice properties
 • a player-level anti-coalition mechanism
 • diverse player input
More in the Future

• Publish the game on Facebook within the
  next one to two months.
• Study the different tagging behaviors of the
  blocker and the couples.
• Apply the competitive mechanism to more
  general human computation game design.
Q &A

More Related Content

More from Academia Sinica

GamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming SystemGamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming System
Academia Sinica
 
Identifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis ApproachIdentifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis Approach
Academia Sinica
 
Improving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user TrustinessImproving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Academia Sinica
 
A Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation SystemsA Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation Systems
Academia Sinica
 
Network Game Design: Hints and Implications of Player Interaction
Network Game Design: Hints and Implications of Player InteractionNetwork Game Design: Hints and Implications of Player Interaction
Network Game Design: Hints and Implications of Player Interaction
Academia Sinica
 
Game Traffic Analysis: An MMORPG Perspective
Game Traffic Analysis: An MMORPG PerspectiveGame Traffic Analysis: An MMORPG Perspective
Game Traffic Analysis: An MMORPG Perspective
Academia Sinica
 

More from Academia Sinica (20)

On The Battle between Online Gamers and Lags
On The Battle between Online Gamers and LagsOn The Battle between Online Gamers and Lags
On The Battle between Online Gamers and Lags
 
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework
Quantifying QoS Requirements of Network Services: A Cheat-Proof FrameworkQuantifying QoS Requirements of Network Services: A Cheat-Proof Framework
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework
 
Online Game QoE Evaluation using Paired Comparisons
Online Game QoE Evaluation using Paired ComparisonsOnline Game QoE Evaluation using Paired Comparisons
Online Game QoE Evaluation using Paired Comparisons
 
GamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming SystemGamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming System
 
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic ApproachAre All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
 
Identifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis ApproachIdentifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis Approach
 
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay NodesToward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
 
Inferring Speech Activity from Encrypted Skype Traffic
Inferring Speech Activity from Encrypted Skype TrafficInferring Speech Activity from Encrypted Skype Traffic
Inferring Speech Activity from Encrypted Skype Traffic
 
Game Bot Detection Based on Avatar Trajectory
Game Bot Detection Based on Avatar TrajectoryGame Bot Detection Based on Avatar Trajectory
Game Bot Detection Based on Avatar Trajectory
 
Improving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user TrustinessImproving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
 
A Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation SystemsA Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation Systems
 
Tuning Skype’s Redundancy Control Algorithm for User Satisfaction
Tuning Skype’s Redundancy Control Algorithm for User SatisfactionTuning Skype’s Redundancy Control Algorithm for User Satisfaction
Tuning Skype’s Redundancy Control Algorithm for User Satisfaction
 
Network Game Design: Hints and Implications of Player Interaction
Network Game Design: Hints and Implications of Player InteractionNetwork Game Design: Hints and Implications of Player Interaction
Network Game Design: Hints and Implications of Player Interaction
 
Mitigating Active Attacks Towards Client Networks Using the Bitmap Filter
Mitigating Active Attacks Towards Client Networks Using the Bitmap FilterMitigating Active Attacks Towards Client Networks Using the Bitmap Filter
Mitigating Active Attacks Towards Client Networks Using the Bitmap Filter
 
An Analysis of WoW Players’ Game Hours
An Analysis of WoW Players’ Game HoursAn Analysis of WoW Players’ Game Hours
An Analysis of WoW Players’ Game Hours
 
Game Traffic Analysis: An MMORPG Perspective
Game Traffic Analysis: An MMORPG PerspectiveGame Traffic Analysis: An MMORPG Perspective
Game Traffic Analysis: An MMORPG Perspective
 
An Analytical Approach to Optimizing The Utility of ESP Games
An Analytical Approach to Optimizing The Utility of ESP GamesAn Analytical Approach to Optimizing The Utility of ESP Games
An Analytical Approach to Optimizing The Utility of ESP Games
 
The Impact of Network Variabilities on TCP Clocking Schemes
The Impact of Network Variabilities on TCP Clocking SchemesThe Impact of Network Variabilities on TCP Clocking Schemes
The Impact of Network Variabilities on TCP Clocking Schemes
 
Automatic Storytelling in Comics: A Case Study on World of Warcraft
Automatic Storytelling in Comics: A Case Study on World of WarcraftAutomatic Storytelling in Comics: A Case Study on World of Warcraft
Automatic Storytelling in Comics: A Case Study on World of Warcraft
 
Game Bot Identification Based on Manifold Learning
Game Bot Identification Based on Manifold LearningGame Bot Identification Based on Manifold Learning
Game Bot Identification Based on Manifold Learning
 

Recently uploaded

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 

Recently uploaded (20)

Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 

KissKissBan: A Competitive Human Computation Game for Image Annotation

  • 1. KissKissBan A Competitive Human Computation Game for Image Annotation Chien-Ju Ho*, Tao-Hsuan Chang+, Jong-Chuan Lee+, Jane Yun-jen Hsu+, Kuan-Ta Chen* *Academia Sinica +National Taiwan University
  • 4. The ESP Game • Coalition problem • partner with themselves • unified strategy • Limited diversity • easy words first
  • 6. Player 1 Player 2 DOG CUTE FLOWER WOOD FLOOR DOG
  • 7. Player 1 Player 2 DOG CUTE FLOWER WOOD FLOOR DOG DELIVER EMOTION BY TYPING THE SAME WORD
  • 8. Couple Couple DOG CUTE FLOWER WOOD FLOOR DOG DELIVER EMOTION BY TYPING THE SAME WORD
  • 9. Blocker Couple Couple DOG DOG CUTE WHITE FLOWER WOOD FLOOR DOG
  • 10. Blocker Couple Couple DOG DOG CUTE WHITE FLOWER WOOD FLOOR DOG KissKissBan A Competitive Human Computation Game for Image Annotation
  • 11. Blocker Couple Couple DOG CUTE DOG Blocked WHITE WOOD FLOWER FLOWER FLOOR The blocked words -5s game time -5s game time 1. First, the blocker has 7 seconds to type words 2. Then, the couples has 30 seconds to type words
  • 12. Blocker Couple Couple DOG CUTE DOG WHITE WOOD FLOWER FLOWER FLOOR Couples win for matching within time limit 1. First, the blocker has 7 seconds to type words 2. Then, the couples has 30 seconds to type words
  • 15. Incentive Structure • Zero sum game • strict competitive • players can not benefit from coalition • Different decision time periods • harder words, longer consideration [Barry et al. 2001] • diverse data contribution
  • 16. Small-scale Experiment • Image Source • 125 images randomly from ESP dataset • Experiment Setup • 17-player gameplay survey • Data collection on Mechanical Turk
  • 17. Gameplay Survey • Fun • 3.76/5 overall rating • 15/17 players claim that would play again • over 60% like to play as a blocker
  • 18. MTurk Experiment • Collecting data from anonymous users • Publish two games for comparison • KissKissBan • ESP Game without taboo words • Each game is played for ~5000 rounds
  • 19. Example of Collected Data ESP ML-KKB BL-KKB beach 3 sea 9 water 3 man 21 man 8 sand 3 beach 10 ocean 3 sea 2 karate 5 black 1 ninja 1 water 1 china 1 kungfu 1 sand 1 ocean 1 ML-KKB: matching between couples BL-KKB: matching between couple and blocker
  • 20. MTurk Experiment • Data property • For each image • ESP: 39.96 labels, 6.56 distinct labels • KKB: 44.17 labels, 11.53 distinct labels • A more diverse player contribution
  • 21. Conclusion • We proposed a competitive human computation game, KissKissBan. • KissKissBan has two nice properties • a player-level anti-coalition mechanism • diverse player input
  • 22. More in the Future • Publish the game on Facebook within the next one to two months. • Study the different tagging behaviors of the blocker and the couples. • Apply the competitive mechanism to more general human computation game design.
  • 23. Q &A