The Cornhole Experiment: A Workshop for Teaching Randomized Trials

In recent years, UK and US governments have increasingly turned to randomized trials to test new policies, most recently in the case of the White House executive order on behavioral science. In the meantime, economics has been undergoing what Angrist and Pischke have called a "credibility revolution," that focuses on offering causal explanations to questions. This has especially been the case in development economics, where "randomistas" have grown in prominence after an influential and controversial 2004 study on the effect of deworming on school attendance in Kenya.
Outside of the work of researchers, expert consultants, and A/B testing in web design, field experiments remain rare. Experimental methods are pretty inaccessible, especially compared to other quantitative methods that are part of many people's daily lives like data visualization, spreadsheets, and even machine learning (Google is widely used, and SwiftKey has had over a billion users at this point).
At the Center for Civic Media, we often try to design fun workshop formats to grow data literacy among the people we work with. This week, together with Anneli Hershman and Joshua Cowls (who are taking the Unpacking Impact class that I co-teach), we organized a one-hour, hands-on workshop that introduces participants to randomized trials with human subjects.
To introduce the idea of a randomized trial to students, we used the game of Cornhole, a favorite pastime at U.S. harvest festivals. In this game, players compete to throw beanbags into a hole on an angled platform. Players score by tossing a beanbag through the hole or by landing a beanbag onto the platform.

The hour workshop includes these components:

  1. (15 mins) Introduced the basic ideas of randomized trials:
    • why we create studies
    • inductive and deductive knowledge
    • predictors and outcomes
    • causal and correlational relationships
    • problems in an experiment (threats to validity)
    • who do the results apply to? (generalizability)
  2. (2 mins) introduced the cornhole experiment
  3. (15 mins) conducted a randomized trial with participants
  4. (throughout) collected data from the experiment
  5. (10 mins) analyzed the data in statistical software
  6. (~15 mins) interpreted and discussed the results

How the Cornhole Experiment WorksIn this simple randomized trial, we try to estimate the chances of a person to score on a given throw. The research question is whether throws to a closer cornhole platform have a greater chance of scoring than throws to a further cornhole platform. The further platform is our control, and the closer platform is our treatment.
In yesterday's workshop, we also considered other variables that might be important, such as the height of the player, the institution of the player (MIT or Harvard), and the experience of the player (the number of previous throws that the player had made).

After discussing ways that selection bias might prevent us from estimating the true effect of playing a shorter cornhole without randomization, we randomly assigned players to two different cornhole games with the classic method of slips of paper in a bag (sometimes, choosing a lower quality technique like this makes the principle clearer). In addition to a group assignment, these slips contained ID numbers, which allowed us to establish a norm of anonymization.
Each player was asked to make 9 tosses using a set of DIY beanbags we created by pouring lentils into ziploc bags and taping them closed. In fact, the entire experiment was designed using household items, to emphasize the accessibility of experimental methods.
As players tossed the bags, two "researchers" recorded their height, institution, ID number, and throw results in a Google Spreadsheet.

Analyzing Cornhole Experiment DataAs the experiment concluded, I downloaded a CSV of the Google Spreadsheet to my laptop and loaded it into the statistical software R, where I had already set up a file to analyze the results.
In a series of cross tables and box plots, I showed a series of bi-variate relationships. Out of 99 throws by 11 players, only 1/3 of tosses in the control group successfully scored. In contrast, over half of the tosses by the treatment group had scored in some way. MIT students, who outnumbered Harvard students, also scored much more on average than the Harvard students, and had a shorter mean height than Harvard students. The mean experience value for scoring throws was higher than the mean for non-scoring throws-- perhaps there might be a relationship between experience and the chance of scoring. The mean height for successful throws was also higher than the mean height for unsuccessful ones; perhaps taller people do better at cornhole?
In a series of logistic regressions, we tested the hypothesis that the players of the closer cornhole had a higher chance of scoring than players of the further cornhole. We also tested all of the possible control hypotheses. Here are the results from the final model:

glm(formula = Binary.Score ~ Group + Throw.Number + Height +
Institution, family = "binomial", data = throws)

Deviance Residuals:
Min 1Q Median 3Q Max
-1.6450 -0.8935 -0.5725 1.0773 2.0173

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -6.74722 7.22492 -0.934 0.35036
TREATMENT 1.39496 0.53187 2.623 0.00872 **
Throw.Number 0.16894 0.08820 1.915 0.05544 .
Height 0.07421 0.11123 0.667 0.50464
MIT -0.23477 0.54951 -0.427 0.66922
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 132.75 on 98 degrees of freedom
Residual deviance: 117.61 on 94 degrees of freedom
AIC: 127.61

Number of Fisher Scoring iterations: 4
In this experiment, assigning a player to a closer cornhole platform had a strong positive effect on the estimated scoring chances of a throw by that person, holding everything else constant. Here, at a cutoff of p<0.05, there was no statistically significant relationship between the estimated scoring chances of a throw and the university, height, or experience of the player (nor in the random intercepts model I later tested). In the control group, a 62 inch tall Harvard student's first throw had an estimated 14% chance of scoring, while a similar student in the treatment group had an estimated 56% chance of scoring.
Interpreting Cornhole Experiment ResultsWe had hoped to conclude the session with an in-depth discussion of the following questions:

  • How much can we trust this result to apply in the future? (p-values)
  • Will this result apply to other places, people, or times? (replication, sampling, generalizability)
  • Should we actually move the cornhole closer, based on this result? (policy uses of experiments)
  • Is this experiment research? How might we change the study to make a contribution to knowledge? (framing research questions)

Unfortunately, class was running late, so we only had 45 minutes for the workshop, and discussion of these wider questions got squeezed out.
Evaluating The Cornhole Experiment WorkshopI feel pretty good about the Cornhole experiment as a classroom workshop, especially as a way to introduce students to experimentation as a method of evaluating their research.
Cornhole has the advantage of being easy to build, fun to play, and comprehensible as an experiment. Because it involves scores or even prizes, it offers a great context for debating the ethics of the experience (since it might disproportionately benefit one group over an other). The cornhole experiment also invites a great debate about what the policy outcomes should be -- are higher scores actually better? Are scores even the right outcome to measure when making decisions about a competitive game?
Statistically, cornhole invites discussion about the reasons for randomization, since everyone knows what it's like to be passed over from a team because they aren't considered athletic. Because distance from the cornhole platform does have a strong relation with the chances of scoring, the workshop facilitator can confidently adjust the treatment depending on whether they want a statistically-significant effect or not. Throws are fast, and with two cornholes, it's possible to collect large numbers of observations in a short period of time. The experiment is also open to a wide variety of covariates, including interesting/spurious ones (university) and an experience predictor that is sometimes statistically significant at p<0.05 and sometimes not.
The cornhole experiment is easily extendable. For example, students suggested that an alternative experiment could allow the treatment group to practice, while not allowing the control group to practice. Another suggestion was that the treatment group might be given positive feedback, while others might be on their own.
Finally, the score data can be reformatted for a variety of modeling approaches. By creating a scale from 0 (no score) to 0.5 (on platform) to 1 (in the hole), it's possible to use linear regression. In this case, we used logistic regression. It's also possible to use count models on a collapsed dataset with one row per person. Finally, it's also possible to fit multilevel models on the cornhole data to account for variation within a player's experience, even while accounting for differences between players who are randomly allocated to different groups.
Next Steps for the Cornhole ExperimentI'm planning to keep running and refining this workshop over the coming months, so keep a lookout!
As experiments become more common in our lives, it is important for the public to understand how randomized trials work so they can make sense of what it means to be in an experiment, how to interpret the results, and run their own. The cornhole experiment is one small step towards a future where running and interpreting an experiment is as simple as playing a backyard game.
How do you introduce your students to experimental methods? I would love to hear your thoughts!
Anneli Hershman; Joshua Cowls; randomized trial; field experiment; cornhole; S61; statistics; AB Test;