PRINT
EMAIL
COMMENT
BUZZ UP
DIGGEXECUTIVE POLL
Bernard Madoff, convicted of running an $65 billion Ponzi scheme, was sentenced to 150 years in jail. What’s your take on his punishment?
OUR PREMIUM CONTENT
The Kiplinger Agriculture Letter
The Kiplinger California Letter
Kiplinger's Biofuels Market Alert
Kiplinger's Retirement Report
SUBSCRIBER HELP
CURRENT LETTER
Overhauling
Financial Regs
PREVIOUS ISSUES
PARC, known as one of the most innovative commercial research labs in the world, was once the in-house lab for Xerox and is known for developing such technologies as laser printing and the Ethernet. It is now a wholly owned subsidiary of Xerox that discovers and implements breakthroughs for commercial and government clients.The accompanying article on "crowdsourcing" was written by PARC researchers Aniket Kittur and Ed H. Chi with help from intern Bongwon Suh.
By, Aniket Kittur, Ed H. Chi, & Bongwon Suh
PARC
Surveys, focus groups, usability studies -- all are ways that companies figure out how to improve their products before they are put on the market. But each method has its own limits and most are costly. "Crowdsourcing" through the Internet may change that.
Crowdsourcing through "micro-tasking" -- where many users perform tasks that take just minutes to perform for a small reward -- is already widely used for certain types of jobs, according to researchers at the groundbreaking Palo Alto Research Center (PARC). "Micro-task markets have been highly successful in domains such as question-and-answer matching (e.g., Yahoo! Answers, Amazon Askville). They offer the practitioner a way to quickly access a large user pool, collect data and compensate users with micro-payments."
Using a service created by Amazon to collect user input at a low cost called Mechanical Turk (mturk.com), the researchers conducted experiments to assess how well and how usefully "turkers," as the participants call themselves, could evaluate products. It had the network critique Wikipedia articles and compared the results to critiques by well-versed Wikipedia administrators. The first effort was disappointing, largely because many responses were attempts to game the system -- get paid without actually reading or seriously evaluating the articles. But when the researchers created questions that could not be answered without reading and thinking about the article, the results were much closer to those of the experts.
"The match to expert ratings is somewhat remarkable, given the major differences between the turkers and the admins," the researchers write. "The correlation between the two populations supports the utility of using crowds to approximate expert judgments in this setting." Their paper spells out the pitfalls and promise of crowdsourcing and offers tips for designing tasks to provide the most reliable results. The top concern, they say, is to have "explicitly verifiable questions as part of the task." That not only helps to ensure genuine answers, but signals that answers will be scrutinized carefully, "which may play a role in both reducing invalid responses and increasing time-on-task."
E-Mail Article
Print Article
