Conducting Online Research on Amazon Mechanical Turk and Beyond

Conducting Online Research on Amazon Mechanical Turk^® and Beyond, written by Leib Litman and Jonathan Robinson, provides both students and experienced researchers with essential information about the online platforms most often used for social science research. This insightful and accessible text answers common questions like, “How do I maintain data quality in online studies?,” “What is the best way to recruit hard-to-reach samples?” and “How can researchers navigate the ethical issues that are unique to online research?” Drawing on their experiences as the founders of CloudResearch (formerly TurkPrime), the authors provide information that guides new users planning their first online studies and engages even the most experienced researchers with detailed discussions about the challenges of online research. The book begins with an overview of Amazon’s Mechanical Turk and its rapid rise within academic research. Then, the authors describe how to set up an MTurk study with screenshots that walk readers through the steps of creating an account, designing a study, collecting data, and using third-party applications to enhance MTurk’s functionality. Later chapters provide readers with a detailed understanding of the MTurk environment and use data from hundreds of thousands of participants and tens of millions of completed tasks to dive into issues like participant demographics, sources of sampling bias, and the generalizability of findings from MTurk. Finally, the book explores the benefits of using other online platforms as a complement to MTurk and the ethical issues that are unique to conducting research with online participant platforms. Throughout the book, the authors share hands-on advice and best practices, such as those for conducting longitudinal studies or carrying out complex studies. Altogether the mix of data, insight, and advice make this book an essential resource for researchers who want to understand the online environment and the most effective ways to conduct research online. 

Jonathan Robinson has been creating software since he was 12 and started his first software company with a few high school friends at age 15. He continued to develop his software skills and apply his expertise to more complex problems as he completed his BA at Queens College, and his MS and PhD with a dissertation in the field of robotic vision and digital curve fitting while at the City University of New York. Jonathan conceived of using Amazon’s Mechanical Turk for participants before there was public academic discussion of its application to research. He designed and implemented the core CloudResearch.com (previously TurkPrime) system together with his coauthor, Leib Litman. CloudResearch was the first cloud-based participant management system that successfully adapted Mechanical Turk for research by designing the features researchers need and managing the complexity that may get in the way of collecting high-quality data.

Jonathan is the co-CEO and chief technology officer of CloudResearch. He is also currently an associate professor and chair of computer science at Lander College, a division of Touro College. In that capacity, he mentors, inspires, and educates the next generation of software engineers and introduces them to his passion for melding cutting-edge technologies with solving real-world problems in the areas of academic and corporate research, digital image computation, approximation, and web application engineering.

Leib Litman received a PhD in experimental psychology from the City University of New York. His early research was in the area of implicit learning, focusing on the relationship between conscious and unconscious information processing. After receiving his PhD, he was a postdoctoral fellow and research scientist in cognitive neuroscience at New York University, where he studied episodic memory using fMRI. Leib’s current methodological research interests are in the area of online data collection, especially in the application of online data collection tools to complex research designs, such as dyadic studies, experience sampling, intensive longitudinal designs, qualitative video and focus group interviews, and global data collection.

Leib is currently an associate professor of psychology at Lander College in Queens, New York, and director of research at CloudResearch.com (formerly TurkPrime). Leib lives in Riverdale, New York, with his wife, Sarah, and their two girls, Bella and Rebecca.