Jeffrey Heninger, 22 November 2022
AI Impacts is a analysis group with seven staff. From Oct 31 – Nov 3, we had a piece retreat. We determined to attempt utilizing Manifold Markets to assist us plan social occasions within the evenings. Listed here are some notes from this experiment.
Construction of the Experiment
Katja created a bunch on Manifold Markets for AI Impacts, and an preliminary assortment of markets. Anybody might add a market to this group, and 5 of us created no less than one market. Every of us would fee every night from 0 to 10 on an nameless Google type. A lot of the questions within the group had been concerning the outcomes of the shape, usually conditional on what exercise we’d try this night. For instance: “On the primary day that no less than 4 individuals start a sport of One Evening Werewolf on the AI Impacts retreat, will the common night score be above 8?” The markets would resolve in some unspecified time in the future the following morning after we had submitted our varieties and Katja calculated the common night score.
Disagreements concerning the Experiment
There have been a number of disagreements about how the experiment was imagined to be run.
Initially, the position of the night score type was unclear. Was it asking on your sincere evaluation of the night or was it a part of the sport? “What quantity would you wish to assign to the night?” is totally different from “How good was your night truthfully?” We determined that we needed sincere responses. Even then, the numbers had been ambiguous. What constitutes a 7 night vs. a 9 night? Completely different individuals’s baselines lead to totally different scores, which might alter the common. After the primary night, we had a greater estimate of the baseline. Lots of the markets had used a median rating of above 8, which was increased than the baseline. This made the markets really feel much less helpful, as a substitute shifting the predictions to decrease chances whereas remaining helpful. It’s not clear why this occurred, but it surely might need been as a result of we didn’t wish to wager in opposition to ourselves having a very good time or as a result of the tail of an unknown distribution is tougher to foretell than the center of the distribution.
One morning, Katja instructed us the common rating earlier than resolving her markets. Zach used this data to wager on these markets. Rick thought that it was unclear whether or not this must be allowed, as a result of not everybody was there and since the earlier dialogue about sincere rankings steered that we must always ask earlier than doing one thing which may give a bonus impartial of prediction capacity. We determined that this could not be allowed sooner or later, and that we’d not inform one another the outcomes of the markets earlier than resolving them.
Unrealized Potential Issues
We considered a number of different potential issues that didn’t find yourself being a problem.
One potential concern was that the interaction between the dynamics of the market and social occasions would possibly make the socialization worse. Somebody who had wager in opposition to having a very good night might need much less cause to need the night to be pleasant to himself and others. If individuals hung out in the course of the night desirous about and steadily betting on the markets, it’d disrupt the continuing actions. In apply, whereas individuals did wager on the markets within the night, it didn’t disrupt the opposite actions.
We had a number of different concepts for mess up the markets: filling out the nameless type a number of instances, colluding or bribing individuals to change their scores, publicly filling out your type earlier than the night begins to control the market, and purposely making an attempt to thwart different individuals’s intelligent methods. None of us tried doing any of those, however they could change into related if the stakes had been increased. There’s additionally the priority that conditional and counterfactual predictions usually are not the identical: For determination making, we want to examine numerous counterfactuals, but it surely’s simpler to make markets that are conditional on us doing one thing. If we determine to do this factor, it’s most likely as a result of no less than a few of us wish to do it, so the conditional prediction might be increased than the counterfactual prediction.
What We Did within the Night
The aim of the markets was to assist us plan out social occasions within the evenings. If the market thought that the night’s score could be extra more likely to be increased if we wore halloween costumes than if we used the recent tub, then we must always determine to put on halloween costumes.
Folks largely didn’t use the markets to determine what to do. On the primary night, the best rated exercise was a guitar sing-along. We didn’t find yourself doing that on any of the evenings. The exercise that appears to have been essentially the most enjoyable for the most individuals was cooperative round-the-table ping-pong. This was executed spontaneously, including extra individuals as they got here to the desk, with none market predicting the end result. We spent a good period of time simply sitting round speaking to one another, which additionally didn’t have a market. Our determination making course of appeared to be much less formal: somebody would recommend an exercise or say that they’d personally do the exercise, and different individuals would be a part of. Having somebody have a look at the markets and announce which exercise rated the best would have added extra steps and group in comparison with what we did.
We additionally tried various the construction of the markets to see if that made them extra helpful. For instance, the market “Will we use the recent tub and have enjoyable tonight?” had 4 decisions for the mixtures of whether or not or not no less than 4 individuals would use the recent tub and whether or not the common night score could be above or under 7. Katja did use this market to argue that individuals ought to use the recent tub.
There appears to have been a couple of issues that stored the markets from being extra helpful: (1) Most of us didn’t know what sorts of social actions many of the remainder of us most popular, so it was exhausting for anybody to make an knowledgeable wager. It wasn’t clear how the market supplied extra data than if we had used a voting system. (2) The connection between 4 individuals doing an exercise and the common night score was too weak for a lot of a sign to undergo. The rankings ended up being noisy, and never particular sufficient for specific actions. (3) The act of checking the markets and saying a choice was extra formal than our precise determination making course of. The market solely included a brief listing of prospects and didn’t recommend spontaneity.
Conclusion
Having prediction markets for the night social actions was a enjoyable addition to the AI Impacts retreat. There have been about 20 markets concerning the retreat which most people on the retreat wager on. However the markets didn’t find yourself having a major affect on what we did in the course of the night.
Most of us didn’t have expertise utilizing prediction markets earlier than the retreat. We determined to not use the markets to make essential choices, as a result of we didn’t know what issues they’d trigger. The markets would doubtless have been extra impactful if we had been extra skilled and if the questions had been about extra essential choices. If we did use the markets for essential choices, we must guarantee that the markets are tougher to use and have extra guidelines and fewer norms governing how we’d wager on the markets.
Because the retreat, Katja has used a market to assist plan an AI Impacts dinner. We plan to proceed experimenting with utilizing prediction markets to make predictions sooner or later.