Q&A with December 2016 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about her experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I work in the field of People Analytics, which combines data science and organization science to help companies manage the people side of their businesses in more evidence-driven ways. I have an MA in Industrial Relations, and have worked in the HR and talent area for 20 years -- but I also am almost finished with an MS in Analytics, and have spent the last 5 years growing deeper in the data space as well.

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

I heard about it following an event I attended at Harvard, I believe it was at the IACSS. I do a lot of survey work in my day job, and have had to design research in my own graduate work -- and I know how critical it is to get decent data for experiments, especially in social science research. It seemed like a good way to give back for all the cool seminars and lectures I've attended (for free) over the years at Harvard.

3. Why have you continued to participate in experiments on DLABSS?

Frankly, it is almost a superstition thing for me -- I feel like if I don't keep participating, my own survey efforts will start to fail, like some kind of cosmic payback. 

4. What is one thing you have learned from one of the experiments you participated in?

I'm always intrigued to see what people are working on, and how they are setting up their studies to get at their research questions. One of the more memorable ones had to do with implicit racial bias, which is something I think about a lot in my work, and it was interesting to participate in the experiment to see how the researcher was trying to test this construct.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

I like the open quality to it, and the fact it feels like you are helping to advance research on important social issues. My hope is that it is a great resource for the researchers as well.

6. What about DLABSS do you think could be improved for experiment participants?

The quality of the studies seems to vary a bit, as does the length -- some experiments seem to involve a much bigger time commitment than others, and I have wondered occasionally whether there is some kind of peer review process prior to releasing the studies. Not often, but once or twice I felt like the construction of the study was pretty poor, and should have gone through another round of edits before it was shared.

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

It's fun! Well, nerdy fun, at least. It doesn't take a lot of time, and it's a neat way to participate in what's happening at the university level in the social sciences, and give back. Anyone who has ever had to develop their own experimental research study should definitely support this effort!

Q&A with November 2016 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I'm a retired municipal worker.

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

A friend of mine said he was participating, and suggested I do the same. I read a lot of research, and I thought it would be interesting to actually participate in research. 

3. What is one thing you have learned from one of the experiments you participated in?

I learned more about my political views after one experiment. Where I thought I was fairly nuanced, I was more binary.

 4. What about DLABSS do you think could be improved for experiment participants?

I wish I had some advice for DLABSS, your site or the performance of the experiments. It's not my field of expertise. I found everything straightforward, which is the best compliment I can give to anyone designing an online experiment.

5. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

The experience is very low-key. Experiments can be ignored, but for the most part, experiments show me more about how I think and view the world.

A Geographical Display of Over 7,000 DLABSS Community Members

This past week, Harvard Digital Lab for the Social Sciences surpassed 7,000 participants. This means that if you have participated in at least one of our surveys, you are a part of a panel consisting of over 7,000 unique individuals from all over the world that have all contributed to social science research! We believe current participants and people considering becoming a DLABSS participant for the first time may like to know more about where those in the DLABSS community are from.

 In fact, DLABSS has participants from over 90 different countries! The map below displays all of our participants from around the world. While a vast majority of our participants are from the United States, there are many people who have joined DLABSS from Europe as well.

Focusing on the United States, the DLABSS community has participants from all 50 states and the District of Columbia, and a total of 1,594 cities and towns across the United States are represented.

We present a map of Europe because this area has the highest concentration of participants outside of the United States. Most of our European participants come from the UK, Netherlands, and Germany.

We are proud to have such a wide-ranging community and hope that our participants are proud to be a part of a diverse panel as well. We realize how important this is for social science research, and hope to expand into geographic areas that appear to be underrepresented in order to increase the diversity of the DLABSS community.

 

  

Benchmarking DLABSS: Replicating Classic and Contemporary Experiments

Recently, we compared the DLABSS volunteer pool to online samples from MTurk and nationally representative sample from ANES and CPS. Building off our earlier analysis, we further benchmark the DLABSS panel by replicating the same three experiments replicated by Berinsky et al (2012) to evaluate the effectiveness of MTurk as a social science research tool.

 The three experiments include an analysis by Rasinski (1989) that investigates the effects of question wording on people’s preferences for government spending, the “Asian Disease Problem” reported by Tversky & Kahneman (1981) that examines the effect of framing on preferences, and Kam and Simas’s (2010) investigation of the effect of an individual’s risk orientation on their preference for certain policy choices. Below we present a table for each experiment that compares the replicated results in DLABSS to the original experiment and the MTurk replication results.

We begin with Rasinski (1989), who finds that when asking people if they think the government is spending too much, too little, or about the right amount on welfare vs. on assistance to the poor significantly changes how one responds to the question. Specifically, people are more inclined to support increased spending when it is phrased as “assistance to the poor” compared to “welfare.” Table 1 below shows the percent of people within each group that believe too little is being spent on these programs.

Framing effects on preference for redistribution: DLABSS, GSS, and MTurk

table1.png

 

The replication performed by DLABSS is in line with the findings of Rasinski and the Berinsky et al MTurk replication. All experiments show a significant difference between the “poor” and “welfare” groups, with the poor group always suggesting too little is being spent more than the welfare group. The DLABSS replication poor group matched the original experiment’s poor group exactly, while the DLABSS welfare group was higher than both of the other platforms. It is possible that slight shifts in public opinion since the original experiment result in this difference.

Second, we replicate the well-known Asian Disease Problem popularized by Tversky & Kahneman (1981), who present two different groups with the problem of a “rare Asian disease” threatening their country and suggests two possible programs to deal with the disease. In brief, the authors find that respondents primed with a “die” frame that describes policy options in terms of deaths (rather than lives saved) are more likely to choose probabilistic (rather than certain) outcomes. Table 2 below shows the percent of respondents that prefer the certain outcome for each frame in DLABSS, MTurk and the original study.

The Asian Disease Problem

table2.png

Across all platforms, people in the positively framed group prefer the certain outcome to the probabilistic outcome, and vice versa for the negatively framed group. The original experiment displays the strongest difference in groups, while the online laboratories find slightly smaller but equally significant effects.

Lastly, we look at a more recent experiment by Kam & Simas (2010), which finds that the amount of risk that people are willing to accept affects their preferences for different policies. Specifically, those who are willing to accept higher amounts of risk are more likely to support probabilistic policy outcomes. Table 3 below displays the results of probit regressions for the three different platforms. We present only the sign and significance for each coefficient for simplicity, but note that the coefficient magnitudes and levels of significance are nearly the same across platforms.

The Effects of Risk Acceptance and Framing on Policy Preferences

Overall, we find that the DLABSS panel is able to successfully replicate all three experiments replicated using MTurk by Berinsky et al. with no significant or alarming differences. Our findings bode well for the future of DLABSS, and online volunteer labs in general, as an effective social science research platform. 

Q&A with August 2016 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I'm a veterinary assistant and college student. I'm planning on becoming a veterinary technician and entering the clergy in my religion. I have an array of strange pets and am two years into my study for priesthood.

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

Someone linked me to a survey a friend was involved in somehow. It's been so long I'm not even 100% what it was.

3. Why have you continued to participate in experiments on DLABSS?

They're pretty neat and I like helping people out.

4. What is one thing you have learned from one of the experiments you participated in?

My ability to recognize faces is much worse than I fully realized. But I'm also getting a bit better at it, thanks to the amount of these that ask you to keep in mind faces.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

The interface is amazing. I've never had an issue with my browser crashing or the page reloading on me or accidentally thinking something was something else and having to redo an entire page.

6. What about DLABSS do you think could be improved for experiment participants?

The experiment categories section could be improved. It's not difficult to navigate or anything, I just think the categories themselves could be reworked a bit to make it easier on participants.

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

Pretty great. By participating you're taking a very small amount of your time to help out other people and have the chance of winning some cash.

Comparing DLABSS to other survey platforms and nationally representative samples

Can low-cost online laboratories substitute for traditional survey platforms? The rise of Amazon MTurk and other online survey tools have helped lower the costs to social scientists conducting experimental and other survey-based studies. Recent findings suggest that MTurk, in terms of subject pool demographics, is broadly comparable to the American National Elections Study (ANES) and Current Population Survey (CPS), often considered the “gold standard” for convenience samples.

Unlike all of these platforms, the Harvard Digital Lab for the Social Sciences (DLABSS) is completely voluntary, meaning research subjects are not directly financially compensated for completing surveys. One question that immediately arises in this volunteer-based setting is what types of people select into DLABSS. How does the DLABSS volunteer community compare demographically to respondents in each of the above survey modalities? To address this question, we collected key demographic data on the DLABSS volunteer pool and compared it to results from MTurk, ANES, and CPS.

The results are presented in Table 1 below. Standard errors are reported in parentheses.

Broadly speaking, while the sample obtained from DLABSS is only a fraction of the overall volunteer pool, respondents are very similar to those from both online and offline survey platforms. On average the DLABSS sample is demographically closer to ANESP (a 2008-2009 ANES panel study) for gender, education, age, marital status and housing status than MTurk, another digital survey platform. On the other hand, DLABSS respondents were on average less wealthy than MTurk workers. In addition, it appears this DLABSS sample may be slightly regionally biased towards the northeast part of the United States.

In addition, we find that DLABSS includes a lower share of white respondents relative to all other survey platforms considered here. We think greater representation of non-majority demographic groups—across race as well as political ideology-- is a potentially promising avenue for volunteer labs such as ours which are able to structurally tailor the underlying demographic composition of the sample pool .

In the coming weeks, we will publish a series of blog posts that build off these descriptive results and further explore the promises and limitations of volunteer labs as a social science tool.

Q&A with July 2016 DLABSS Gift Card Winner

wild.jpg

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I am a meat cutter for a grocery chain in Tennessee.

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

Reddit; something to do.

3. Why have you continued to participate in experiments on DLABSS?

They are interesting and get me thinking.

4. What is one thing you have learned from one of the experiments you participated in?

Politics is confusing.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

Easy to use.

6. What about DLABSS do you think could be improved for experiment participants?

Wordiness of some questions.

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

It's a great way to think about current issues and fill in some time with a fun survey.

Q&A with June 2016 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I work as a caregiver for an adult with autism here in Oregon. I grew up in Oregon and my job is a great way to feel like I'm helping people but also allows me to explore the state from a unique perspective which is a great experience.

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

I don't remember exactly how I found DLABSS, but I assume it was a link somewhere online or possibly mentioned in the article about the results of a study. I decided to particapte for the first time because it seemed interesting. Besides, I enjoy filling out surveys, it gives me a small sense of accomplishment.

3. Why have you continued to participate in experiments on DLABSS?

In the past I've taken surveys in a vain attempt to make some extra money, but I continue to participate in DBLASS because I feel like I'm providing some value to collective knowledge even if there is no personal reward. Mostly I've continued to participate because they continue to be interesting.

4. What is one thing you have learned from one of the experiments you participated in?

I think the biggest thing I've learned is that I'm incredibly over confident in predicting what direction a set of questions is leading. Every time I guess what the purpose of a set of questions was I always turn out to be wrong.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

The best feature of DLABSS is that it opens up social science research to a group of people that are often entirely missed when research is conducted using college students. The participant pool is still self-selected, but it's a different self-selected group and that's probably a good thing.

6. What about DLABSS do you think could be improved for experiment participants?

The website seems to have some minor compatibility issues when running Linux, but that's really a minor quibble.

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

I would describe the experience as generally pretty good. It's an opportunity to do something that's both interesting and potentially helpful without it actually being particularly difficult which is rare in life.

Q&A with May 2016 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I'm from Uruguay, I am wrapping up my bachelor's degree in Sociology and I plan to do a masters in cognitive science here as well. After that, I would like to apply to a Ph.D abroad. I'm also full time employed so I always steal hours from sleeping!

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

I saw a Facebook post by Gary King, I clicked on the link and it was an in depth column about DLABSS, it was fun, it seemed really outside of the box thinking, so I wanted to participate providing the most faithful data I could.

3. Why have you continued to participate in experiments on DLABSS?

Whenever I see a study that I fit the criteria I delve right into it, I find it rewarding to collaborate on cutting edge research.

4. What is one thing you have learned from one of the experiments you participated in?

That online based studies and experiments are here to stay, also, that the whole methodology is still taking its first steps so a lot of thinking and fine tuning has to go into it. It's a massively powerful tool that can provide insights into several different phenomena that normally were studied with geographical constrictions.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

Its easiness to use and how well-thought the experiments and studies are.

6. What about DLABSS do you think could be improved for experiment participants?

I think more studies focusing on cross-national and cross-cultural differences, that way more people would feel engaged. Another cool thing would be to be emailed some sort of summary after the studies are published.

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

It's extremely fun!

Effective Samples in Online Experiments: A Role for Volunteer Labs?

Earlier this month Science Magazine noted the potentially worrisome dependence of psychologists on Amazon.com's Mechanical Turk (MTurk) for experimental research subjects. Online labor markets, and MTurk in particular, have equipped social science (and many other academic and non-academic fields) with a low-cost, often reliable source of survey participants. "Social science for pennies" has caught on quickly, reflected by the rapid proliferation of academic publications using data from MTurk. While not everyone agrees about exactly how it should be integrated into social science, MTurk is generally perceived as an enormous and diverse subject pool for academic research.

So what's the cause for concern? As the Science piece points out, researchers using methods imported from wildlife ecology previously found that while MTurk boasts a participant pool of over half a million subjects, its effective sample is just 7,300. This essentially means that when studies are advertised on MTurk, only a small fraction of the potential pool is likely to participate. 

 This is obviously problematic if one of MTurk's most attractive features is its purported ability to provide diverse and representative samples. While certainly an extreme scenario, it would be strange and unsettling if most of the thousands of studies now being published using MTurk all relied on the same data points in the population. Small effective sample may be related to financial compensation incentives on behalf of Turkers--while thousands upon thousands of people may initially sign up, only a subset find Human Intelligence Task (HIT)'s involving social science surveys lucrative and desirable enough to scale up and participate repeatedly.

This one area in which, by design, volunteer labs might possess an advantage over MTurk and other paid online labs. Whereas paid subjects may become inactive once they become dissatisfied with compensation or simply find something better to do, volunteer subjects are presumably joining DLABSS or other initiatives for non-monetary purposes. This may create entirely new selection bias issues (that we will address in the coming weeks and months), but also might result in larger effective samples. Indeed, the DLABSS volunteer community is quickly approaching 6,000 people, just short of MTurk's effective sample size estimated in recent years. To date over half of DLABSS volunteers have participated in more than one study, and nearly 40% have participated in more than two. The average DLABSS volunteer has taken 3.7 studies. 

In any case, effective samples are just one of many challenges that social scientists are facing as they pivot to online experimental outlets such as MTurk. It remains to be seen precisely how severe MTurk's issues are, as well as the extent to which volunteer-based labs can help mitigate such problems. 

DLABSS is attempting to help fill these knowledge gaps by comparing its own subjects and their performance with MTurk and other survey modalities across a large range of experimental settings this summer. Of course, even if volunteer labs are cheaper and prove more effective at avoiding effective sample and other biases, we would like to see volunteer labs gradually emerge as complements to other existing survey platforms. For instance, they might help "break ties" when a study obtains different results across in-person and MTurk samples, and could help researchers discover bugs in their experimental research designs.

We plan on releasing a comprehensive summary of our initial findings in working paper during early Fall 2016. Our preliminary assessments suggest that DLABSS volunteers are in fact demographically similar to "Turkers" and that the Lab is able to replicate research findings obtained using other experimental modalities. In the meantime, we will publish snapshots of various findings as they come in over the summer. So stay tuned!

Q&A with April 2016 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

Musician/guitarist, I am 27, male, with an associates in CAD and do some freelancing in the field. 

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

I first heard of the DLABSS surveys through Craigslist, the subject matter seemed interesting and I wanted to see what the surveys were all about.

3. Why have you continued to participate in experiments on DLABSS?

 I continue to take the surveys because they deal with modern occurrences and themes. Some of the topics I have strong feelings towards and enjoy responding. 

4. What is one thing you have learned from one of the experiments you participated in?

How the Unconscious mind influences the conscious and molds judgement and opinion.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

I like how new surveys are sent to my inbox and I can do them all at will. The website also has a user friendly interface. 

6. What about DLABSS do you think could be improved for experiment participants?

Maybe more visual stimulation to appeal to millennials and the like. 

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

DLABBS is a quick and easy way to give your opinion on current events. What ever survey you start with will more than likely cause you to think in a new light and you may learn something new about yourself.

Q & A with March 2016 DLABSS Gift Card Winner

pexels-photo-30342.jpg

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I work in IT management.  I am married with 3 kids and live in the suburbs of Chicago. 

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

I really don't remember.  I think someone sent me a link, the studies always seem to be interesting.

3. Why have you continued to participate in experiments on DLABSS?

Generally because the experiments tend to engage me and not take that much time.  

4. What is one thing you have learned from one of the experiments you participated in?

That I have trouble remembering what I clicked on less than a minute ago. 

 5. In your opinion, what is the best feature of DLABSS and/or its website?

Simplicity.  Not too much fluff or GUI complications. 

6. What about DLABSS do you think could be improved for experiment participants?

Don't make me enter my email address twice.  I think you guys figured out the issue where identical emails with with different capitalization were distinct entries in your database.

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

Interesting without being a burden.

Volunteer Power: Reaching the 5,000 Milestone

When Founding Director Ryan Enos created the Harvard Digital Lab for the Social Sciences (DLABSS), his vision was to offer both a powerful and widely accessible tool for experimental social science. Eighteen months later, DLABSS has grown into a widely used platform among Harvard faculty and students. We have hosted over 50 studies to date in the fields of political science, economics, psychology, sociology and business. See a list of our 26 currently active studies here

Volunteerism is the engine that makes DLABSS go. Our virtual community is sprawling and this week has surged to over 5,000 members, who collectively have contributed to social science over 18,000 times! To all of our volunteers, the DLABSS team sincerely thanks you for making our lab possible. 

We hope you will stay engaged in our virtual community. To that end, moving forward we will strive to host even more diverse and interesting studies that allow participants to learn about the world while helping improve our understanding of it. We see this as a two-way learning process and want to make the DLABSS experience more intrinsically rewarding for everyone involved. If you have comments or suggestions for us, feel free to email us at manager@dlabss.harvard.edu. In the meantime, we will be busy designing studies and features we hope you will find valuable.

Thanks for your contributions to cutting-edge social science. We look forward to growing together!

-The DLABSS Team

Q & A with February 2016 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about her experience working with DLABSS on multiple experiments. We have posted her responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I am a laboratory manager for a psychology research lab, where I am responsible for the recruitment, execution and general oversight of human subject research experiments.

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

I heard about Harvard DLABSS via a craigslist posting. I suppose I participated for the first time because, as a researcher myself, I am naturally interested in research studies.  My being a researcher also lends a kind of empathy for the (often difficult) process of subject recruitment, so I like to contribute when I can.

3. Why have you continued to participate in experiments on DLABSS?

In addition to the reasons mentioned above, the subject matter of the experiments is interesting and the online format makes participation incredibly easy.

4. What is one thing you have learned from one of the experiments you participated in?

I don't know if I could answer this too specifically, although I'm sure I come away from each experiment with some contemplation on both the subject matter and my own responses.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

The clear and simple layout of the website makes it extremely easy to choose and complete studies.

6. What about DLABSS do you think could be improved for experiment participants?

Obviously guaranteed compensation for participation is desirable from a participant perspective, although I'm sure that's unrealistic for the lab. Otherwise, I cannot think of any necessary improvements! 

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

I would describe the overall experience as positive. Compared to other studies that I've participated in, both in person and online, these studies are generally far more interesting. Additionally, the ease of completing the experiments online and on your own time is great- if you have fifteen minutes to kill, why not? 

Q & A with January 2016 DLABSS Gift Card Winner

 

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I have been employed in several different roles for over 20 years at a large technology company that has, at times, made its way way into the Fortune 500.   I am currently serving in an Information Technology role where I help implement solutions for use within our company using software developed for sale by our company

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

I'm honestly not sure.   I have been a participant for a while,  and may have heard about it on Reddit. I always enjoyed introductory Psychology in college and thought that the idea of participating in surveys/experiments would be fun.   The online nature of this experience makes it easy to participate on my own terms.

3. Why have you continued to participate in experiments on DLABSS?

I still retain the same thoughts as when I first started.   I like the idea of being able to contribute in a way that is still convenient for me.

4. What is one thing you have learned from one of the experiments you participated in?

I'm not quite sure how to answer this one.   I tend to focus on providing the most accurate answers from my point of view.   I can say that it has caused me to stop and contemplate things a little more deeply than I might otherwise be naturally inclined to do.

 5. In your opinion, what is the best feature of DLABSS and/or its website?

The entire experience seems polished and well laid out.  I never struggle with presentation or comprehension.   If I have to pick something,  however,  I would have to say the notifications.   That might be a lame feature to focus on, but I wouldn't remain active without them.

6. What about DLABSS do you think could be improved for experiment participants?

Perhaps focus on giving a better idea of my progress throughout the process.  How far along am I?   At my current pace,  when am I expected to be finished? 

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

The surveys have been engaging as opposed to shallow.   The in-depth experience causes you to ponder more than the surveys that I participate in via Google Rewards,  for instance. 

Q & A with December 2015 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about his experience working with DLABSS on multiple experiments. We have posted his responses below:


1. What is your occupation? Tell us some basic background information about yourself.

I'm a student about to graduate with a B.S. in Chemistry and a minor in Geology at the University of Illinois at Urbana Champaign. I conduct research in organometallic catalytic methods. 

2. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

A reddit post a couple years ago. I decided to fill out some of the surveys because it was easy and quick and I know that those responses are valuable data to researchers who sometimes have issues finding enough participants.

3. Why have you continued to participate in experiments on DLABSS?

Same reason I started, it doesn't take much of my time, gives me something to do, and provides valuable data to researchers. Also the chance to win these gift cards is a decent incentive.

4. What is one thing you have learned from one of the experiments you participated in?

People really like to study things that correlate with racial makeup apparently. Lots of surveys about race and sexual orientation and their relations to political opinions and such. 

 5. In your opinion, what is the best feature of DLABSS and/or its website?

Its nice to have the surveys aggregated on a an internal site rather than linking to outside survey sites. Also nice that it vaguely keeps track of the amount of surveys you've completed and e-mails you reminders about new ones. 

6. What about DLABSS do you think could be improved for experiment participants?

I'm not sure, answering surveys isn't a particularly difficult or complex task. I guess keep the website simple and easy to use? 

7. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

Good, it's easy and doesn't take much time and doesn't interfere with my day to day life so there's not any downsides to participating. People need survey data, I'm not doing anything with my 10 minutes, everyone wins by filling out these surveys.

Why We Vote Matters: The Impact of Altruistic Voting on Election Outcomes

This is a guest post by DLABSS Researcher Daniel Mahler. This research is part of a larger Danish research project, the findings of which are very much in line with the results from the US sample discussed below. The findings may apply to other democracies as well.

Extensive research in political science has documented that some voters cast their ballots based on selfish concerns, while others vote for the candidate they think is best for society as a whole.  But does this dichotomy matter for the outcome of elections? And what would happen if some people were to change their moral motivation for voting? Seeking answers to these questions, 400 participants at the Harvard Digital Lab were asked the following three questions:

1.     Who would you vote for if the presidential election were held tomorrow?

2.     Who would you vote for if you were to consider only what is best for yourself?

3.     Who would you vote for if you were to consider what is best for society as a whole?

The participants were also asked to place the 2016 presidential candidates on a political scale from 1 (extremely liberal) to 7 (extremely conservative). The average placements of a selection of the candidates are shown below.

With these placements, it is possible to compare where on this line, on average, individuals place their actual, selfish, and altruistic (i.e. societal) votes. That is, we can see what would happen if people changed why they vote. 

As shown in the figure above, the answer seems to be that if more people were to vote selfishly, right-wing candidates would receive more votes. Conversely, if more people voted altruistically, the outcome would be more left-winged.

We can go one step further and compare the variation or standard deviation in the votes. A large standard deviation signifies less political agreement and greater popularity of candidates on the far ends of the political spectrum. This is illustrated below.

Evidently, if more people voted selfishly, then candidates on the extreme ends of the political spectrum would receive more votes, whereas there would be slightly more political agreement if more people voted altruistically.

So what? Does it matter whether people vote selfishly or altruistically? These results suggest it does indeed matter whether people vote for what they think is best for themselves or society as a whole. If candidates knew this, they might be able to increase their chances of winning elections by influencing the reasons why voters vote.

Reflecting on 2015: DLABSS New Year's Resolutions

As we near the arrival of 2016, the DLABSS Team wants to quickly fill you in on what has been a highly productive 2015! In this post we share how DLABSS has grown in 2015, and introduce some major upgrades on the way next year. As always, we want to thank the many volunteer participants and Harvard researchers from around the world whose efforts are pushing social science forward.

Participant Growth

What we did in 2015: 

The DLABSS pool of volunteer participants is growing faster and more consistently than any time in our short history. As the below graph illustrates, our pool increased by nearly 300% in the last 12 months! We grew from just over 1,000 volunteers in January to well over 3,000 by mid December. Dozens of people are joining the DLABSS community every week.

While we are excited about our yearly growth, even more encouraging is the surge of new volunteers in recent weeks. As you can see from the below graph, weekly volunteer totals have spiked since November. Since mid-November, on average roughly 100 new people have joined DLABSS every week!

 What we are doing in 2016: 

We think this progress is but the tip of the iceberg. We are committed to making DLABSS the go-to resource for Harvard experimental social science. This goal requires an even larger subject pool. To this end, we are launching a brand new website in January 2016 that will make participating in DLABSS easier and more enjoyable for volunteers and researchers alike.

 Volunteer Diversity

 What we did in 2015:

One of the advantages of online survey research is that researchers can attract a wide variety of participants. Ideally a pool of respondents is representative of society. Below we illustrate the breakdown of our volunteer pool in terms of gender and education. As you can see in the below figures, so far women represent a slight majority within our volunteer community, while volunteers have very diverse educational backgrounds.

blog4.png

 

What we are doing in 2016:

 Of course, survey research is not without limitations. Online labs such as MTurk, for example, have been shown to underrepresent minorities and produce samples biased towards younger generations. In 2015 we noticed that the DLABSS volunteer base was exhibiting similar trends. As such, we have formed partnerships with RetiredBrains and Workforce50.com, two senior citizen platforms, and are planning to engage more African American and Latino participants through online marketing. Our hope is that targeted marketing to diverse groups of potential volunteers will make DLABSS even more representative of the general population.

 Moreover, we are currently undergoing rigorous testing of DLABSS as a valid social science tool by using it to replicate several classic and contemporary social science experiments. Similar substantive results, if produced, will help build credibility for DLABSS as a viable resource for Harvard social scientists.

 Experiment Diversity

 What we did in 2015:

 We launched a total of 21 experiments by faculty, post-docs, graduate and undergraduate students from Harvard's Faculty of Arts and Sciences, Kennedy School of Government, and 48% of experiments were primarily political in nature, 19% economic, 19% psychology, and 14% group and race relations. The plot below plots the cumulative number of responses for each survey in 2015, showing that almost all surveys received 200 responses within 3 weeks. The 5 lines in red represent the surveys from November and December 2015, signaling that our most successful surveys have been our most recent.

What we are doing in 2016:

In the short term, we are launching experiments on American politics, healthcare policy, risk acceptance, and personality. In addition, we hope to add more content on a wide range of issues--such as foreign policy, economics, and sports.

Finally, the team at DLABSS would like to thank all of our volunteers and researchers during 2015. Please join us for what promises to be a productive and enlightening 2016. Happy New Year!

Q & A with November 2015 DLABSS Gift Card Winner

The latest winner of our monthly $50 Amazon gift card lottery agreed to answer a few questions about her experience working with DLABSS on multiple experiments. We have posted her responses below:


1. How did you first hear about Harvard DLABSS, and why did you decide to participate for the first time?

A friend of mine told me about DLABSS.  I'm not sure how she found out about it. I thought the topic was interesting and I liked the idea of possibly winning some money.

2. Why have you continued to participate in experiments on DLABSS?

Because they are very interesting.

3. What is one thing you have learned from one of the experiments you participated in?

I learned about new ideas and concepts that I don't usually think about.

 4. In your opinion, what is the best feature of DLABSS and/or its website?

Interesting topics...and I have a chance to win some money!

5. What about DLABSS do you think could be improved for experiment participants?

I can't think of anything. The surveys are pretty straightforward.

6. If you were talking to someone considering participating in DLABSS for the first time, how would you describe your overall experience?

Easy and fun. It doesn't take a lot of time away from your busy schedule to participate.

You Know It When You See It: Measuring Gerrymandering through Visual Perceptions

This post is by DLABSS researcher Aaron Kaufman. Aaron is the William Yandell Elliott Fellow in the Department of Government at Harvard University. His interests include causal inference, field experiments, natural experiments, statistical programming, and natural language processing. 

"Voters should choose their politicians; politicians shouldn't choose their voters." It seems troubling, then, that every 10 years, politicians redraw their district lines to suit their own reelection purposes. In some states, the majority party may draw the lines to strengthen incumbents and hurt opponents' abilities to gain election. In other states, incumbents from both parties conspire to keep themselves in office, effectively preventing newcomers from challenging their political dominance. This process is called Gerrymandering. It is completely legal, and sometimes results in abnormally-shaped districts, like Alabama's 16th State Assembly District:

kaufman1.png

Research in political science has attributed many adverse effects to Gerrymandering. For example, some say it contributes to polarization, which worsens legislative gridlock in both states and in Congress. Others say it extends the incumbency advantage, where leaders in office have a competitive advantage over newcomers, and thereby increases the amount of money in politics.

On the other hand, recent research argues that Gerrymandering has little effect on politics. Since many voters live in ideologically similar neighborhoods anyway, and if most voters in a county are Republican, those voters are best represented by a Republican regardless of the shape of the district. There is a thin line, however, between a district drawn to include like-minded voters and a district drawn to protect politicians from challengers.

This line remains very unclear, but what we know about it comes from the Voting Rights Act (VRA). The VRA lists guidelines for how to legally redistrict cases to avoid Gerrymandering, and one of the key criteria is "compactness". Many political scientists have tried to mathematically define compactness, but none have been very successful. Today, the courts use any of several dozen academic and mathematical measures in trying to determine if a state's districts are Gerrymandered, many of which conflict with each other. All these measures start from a common ground: a district may be Gerrymandered if it is not compact, and we can measure compactness by measuring the population, size, and shape of a district.

In this project, we start from a different premise. Coactness cannot be formalized; rather, it is the kind of thing where you know it when you see it. So we do just that! Using DLABSS, we show people pairs of districts, and ask them which district in each pair they think looks more compact, whatever that means to them. With enough responses, we can rank every district from least compact to most compact. Finally, once we have compactness ranks for every district, we can use cutting-edge sorting algorithms and machine learning to learn the relationship between the mathematical and academic measures of compactness, and what people consider to be truly compact. In summary, we are building a model to predict how Gerrymandered a district is!

Our major finding is that these traditional mathematical measures don't very well capture what people think compactness looks like. Below is a plot of the relationship between our DLABSS-based rank of districts and three major mathematical measures: the Polsby-Popper Test, the Reock Test, and the Schwartzberg Test. Each dot represents a real district. Its horizontal position refers to its rank along a mathematical measure of compactness; its vertical position represents its true compactness.

kaufman2.png

The plots are a complete mess! There is a very weak relationship between the two measures, indicating that the way judges and lawyers and lawmakers interpret the compactness criterion in the Voting Rights Act have little to do with the original meaning of the law. On the other hand, the tools we have developed are based on intuitive understandings of compact shapes. We hope that our research will set a new standard for identifying and correcting Gerrymandered districts.