Can low-cost online laboratories substitute for traditional survey platforms? The rise of Amazon MTurk and other online survey tools have helped lower the costs to social scientists conducting experimental and other survey-based studies. Recent findings suggest that MTurk, in terms of subject pool demographics, is broadly comparable to the American National Elections Study (ANES) and Current Population Survey (CPS), often considered the “gold standard” for convenience samples.
Unlike all of these platforms, the Harvard Digital Lab for the Social Sciences (DLABSS) is completely voluntary, meaning research subjects are not directly financially compensated for completing surveys. One question that immediately arises in this volunteer-based setting is what types of people select into DLABSS. How does the DLABSS volunteer community compare demographically to respondents in each of the above survey modalities? To address this question, we collected key demographic data on the DLABSS volunteer pool and compared it to results from MTurk, ANES, and CPS.
The results are presented in Table 1 below. Standard errors are reported in parentheses.
Broadly speaking, while the sample obtained from DLABSS is only a fraction of the overall volunteer pool, respondents are very similar to those from both online and offline survey platforms. On average the DLABSS sample is demographically closer to ANESP (a 2008-2009 ANES panel study) for gender, education, age, marital status and housing status than MTurk, another digital survey platform. On the other hand, DLABSS respondents were on average less wealthy than MTurk workers. In addition, it appears this DLABSS sample may be slightly regionally biased towards the northeast part of the United States.
In addition, we find that DLABSS includes a lower share of white respondents relative to all other survey platforms considered here. We think greater representation of non-majority demographic groups—across race as well as political ideology-- is a potentially promising avenue for volunteer labs such as ours which are able to structurally tailor the underlying demographic composition of the sample pool .
In the coming weeks, we will publish a series of blog posts that build off these descriptive results and further explore the promises and limitations of volunteer labs as a social science tool.