Paper Reading #6: TurKit: Human Computation Algorithms on Mechanical Turk


Reference
Author: Greg Little(student), Lydia B. Chilton(Computer Science grad student)*, Max Goldman(grad student), Robert C. Miller(Associate Professor)
Affiliation: MIT CSAIL and University of Washington* respectively
Presentation: UIST’10, October 3–6, 2010, New York, New York, USA

Summary
Hypothesis
The paper presents the crash-and- rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. The paper hypothesizes that TurKit allows the programmer to think about algorithmic tasks as simple straight-line imperative programs, where calls to MTurk appear as ordinary function calls.It also hypothesizes that the crash-and-rerun programming makes it cheap to rerun the entire program up to the place it crashed, if our program crashes, since local computation is cheap. Other benefits that were hypothesized include incremental programming, easy implementation, retroactive print line debugging.

Methods
TurKit was used in implementation of various application like iterative writing( one turker writes a paragraph with a goal, and subsequent turkers try to improve upon the paragraph),blurry text( new guesses as to the blurred text's meaning are added and changed) recognition,decision theory experimentation(simulate human decision making in a random guessing scenario) and pyschophysics experimentation. Different case studies were also done to show the results of the application.

Results
Results claim that the programming model is good for prototyping algorithmic tasks on MTurk, and that it sacrifices efficiency for programming usability.Users were unclear about which aspects of a TurKit script were stored in the execution trace, and which parts could be modified or re-ordered while others were unaware of the parallelism feature.
People tend to keep to the original style and formatting while working on paragraphs. In the blurry text recognition the final result was nearly perfect. In the decision theory experimentation TurKit was useful in coordinating the iterative nature of the process, but not necessarily very good at simulating actual human behavior. It was effective in the area of psychophysics experimentation, since calls to MTurk were embedded within a larger application.

Discussion
I feel indifferent about the results and experiment with TurKit. The idea to not execute certain sets of instructions but get to the point of last execution to run the program seems promising but might be limited to some basic forms of programming only. I am not really sure how MTurk works, so I would not really understand the effect the tool will have. As users were not aware of the parallelism feature and never really explored it to fullest, I feel like a lot of useful feedback was left out as the functionality was not tested to its fullest.

Comments

Post a Comment

Popular posts from this blog

Paper Reading #20: The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures

Image
Reference Authors and Affiliations: Leah Findlater,Jacob O. Wobbrock ,The Information School DUB Group University of Washington Seattle, WA 98195 USA Darren Gergle, School of Communication Northwestern University Evanston, IL 60208 USA James J. Higgins,Department of Statistics Kansas State University Manhattan, KS 66506 USA Presentation: CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. Summary Hypothesis: The paper present the Aligned Rank Transform (ART) for nonparametric factorial data analysis in HCI. They propose a preprocessing step that “aligns” data before applying averaged ranks, after which point, common ANOVA procedures can be used, making the ART accessible to anyone familiar with the F-test. They hypothesize that researchers only familiar with ANOVA can use, interpret, and report results from the ART. Contents Conover and Iman’s Rank Transform (RT) uses the parametric F-test on the ranks, resulting in a nonparametric factorial procedu...
Read more

Paper Reading #29 : Usable Gestures for Blind People: Understanding Preference and Performance

Image
Reference Authors and Affiliations: Shaun K. Kane, Jacob O. Wobbrock - The Information School, DUB Group University of Washington Seattle, WA 98195 USA Richard E. Ladner - Computer Science & Engineering, DUB Group University of Washington, Seattle, WA 98195 USA Presentation :CHI 2011, May 7–12, 2011, Vancouver, BC, Canada Summary Hypothesis The paper suggests that the blind people have different gesture preferences than sighted people, including preferences for edge-based gestures and gestures that involve tapping virtual keys on a keyboard. They hypothesize differences in the speed, size, and shape of gestures performed by blind people versus those performed by sighted people. Contents Accessible touch screens still present challenges to both users and designers. Users must be able to learn new touch screen applications quickly and effectively, while designers must be able to implement accessible touch screen interaction techniques for a diverse range of devices and application...
Read more

Paper Reading #27 : Sensing Cognitive Multitasking for a Brain-Based Adaptive User Interface

Image
Reference Authors and Affiliations: Erin Treacy Solovey,Francine Lalooses,Krysta Chauncey, Douglas Weaver,Margarita Parasi,Matthias Scheutz,Robert J.K. Jacob at Tufts University, Computer Science,161 College Ave., Medford, MA Angelo Sassaroli, Sergio Fantini, at Tufts University Biomedical Engineering 4 Colby St., Medford, MA Paul Schermerhorn,Indiana University Cognitive Science Bloomington, IN Audrey Girouard,Queens University School of Computing 25 Union St., Kingston, ON K7L 3N6 Presentation: CHI 2011, May 7–12, 2011, Vancouver, BC, Canada Summary Hypothesis The paper describes two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Authors then present a human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. They hypothesize that the prototype system serves as ...
Read more