There is significant overlap between two types of emerging forecasting systems: prediction markets and elicitation platforms. Prediction markets generate contracts corresponding to various potential outcomes (similar to tickets in horse racing), assign a payoff to owning the contract corresponding with the outcome observed in reality, and facilitate trading of these contracts by participants until the outcome occurs. Elicitation platforms recruit a pool of participants and pose questions. The participants enter probabilities, and the platform aggregates them, in ways ranging from a simple average or median to complex algorithms, to produce the platform’s consensus estimate.

In both cases, the aggregate actions of participants produce an output that can be taken as a probabilistic forecast- the “wisdom of the crowd”- and this information acts as a public good. They are also both considered to be an improvement of how the future is commonly discussed in society: overconfident pronouncements from pundits who know that as long as they are expressing the correct worldview for their ingroup, they don’t have to worry about being held accountable. However, there are also important differences in how these systems work, and these have consequences for evaluating their outputs and participants.

1. Scoring

The foundation of many differences is how these two systems score their participants.

Scoring is very easy on a prediction market: profit! If you buy a contract at 50 cents and sell it for 80 cents, it doesn’t matter what your actual internal probability estimate was, it doesn’t matter how long you’ve held the contract. It doesn’t even matter whether that outcome occurs or not. You made 30 cents (minus any transaction fees) in profit.

On an elicitation platform, the administrator needs to choose a scoring system. Explaining all the concerns at play in this process would easily be its own essay [1], so take it in faith that scoring these questions is very complex, requires many methodological choices, and often results in unintended ways to game the metrics involved. Generally, if an event occurs, forecasts are penalized in some greater-than-linear manner according to their distance from 100% (and if it doesn’t occur, the distance from 0% is used).

Also, on elicitation platforms, all questions are weighted equally- regardless of importance, regardless of duration, often regardless of “difficulty”, and in some cases regardless of whether a forecaster even forecasted on the question (the median forecast of their peers is imputed in this case). The larger set of questions can be broken down into subsets- Metaculus, for example, scores some questions in the “2024” category if they started and ended in 2024, or “2023-2024” if they started in 2023 and ended in 2024, and forecasters could end up with a stronger score in one category than another.

Going back to scoring prediction market participants by profit: this is easy, but it has its own what might be called exploits. The most obvious of these is making a trade contrary to what a trader believes the true probability is, anticipating that there will be a price movement in that direction.

It follows that, particularly if the prediction market has a native feature for commenting (or even if there is a lot of overlap with social networking sites like Twitter), traders are incentivized to “pump” their positions by biasing the information they discuss in comments, or take positions with a strategy of pumping them.

Also, it’s possible to earn modest profits on prediction markets by taking advantage of the high implied time value of money on longer-term questions, such as buying “No” contracts on the question “Will there be a wealth tax by the end of 2024?” for 95 cents at the end of June 2024 and holding them until the end of the year when it presumably resolves “No”.

2. Position sizing

If scoring is very easy on prediction markets, position sizing is very easy for participants on elicitation platforms: either you forecast on a question, or you don’t. Prediction market traders have to consider how many shares they want to buy or sell at available prices, and then constantly evaluate the size of their position over time. The influence of different questions on their score- their profit- differs by orders of magnitude, and they might be happy to limit themselves to only a small portion of the available questions. This skill- and, for that matter, risk tolerance itself- creates another input to success on prediction markets, aside from forecasting ability.

3. Discourse

We brought up commenting features. People on elicitation platforms share more information, and are just nicer. There aren’t any “longs” or “shorts” motivated to have the market price move in a given direction; pretty much everyone is on a continuum between 0 and 1, often inputting their forecast explicitly. Participants are usually admitting up front that they are uncertain. While a prediction market might have one person with a 60% forecast and one with an 80% forecast on opposite sides, in an elicitation platform they are able to start from a place of broad agreement (and even the person with a 60% forecast and the one with a 30% forecast are mutually agreeing that there is a lot of uncertainty).

4. Marketing

Real-money prediction markets have a marketing / participant recruitment advantage because people like to earn money, so they can speculate for profit (or hedge other assets, or arbitrage against other financial markets). This is particularly true regarding speculation, now that more people are being comfortable with sports betting and with ultra-speculative trading in financial markets.

Also, prediction markets- especially those with sufficient participation and liquidity- can tap into the solid theoretical foundation of the efficient markets hypothesis. They have done a better job gathering highbrow endorsements from economists and libertarian thinkers.

5. Public relations

A corollary to the first point raised in the marketing section is that people who find gambling distasteful are increasingly radicalized by its carefree normalization (or what I guess one could call its normalization in the white-collar sphere; the lottery, which offers terrible odds, has been a tolerated way for people to gamble for a long time). An extension to this is that the idea of monetizing elections threatens to tarnish democracy. And of course, when it was revealed that the US government was working on a prediction market for intelligence purposes, there was an outcry from people who believed that weighty life-and-death issues such as terrorist attacks shouldn’t be targets for financial speculation [2]. Also, a prediction market that permits or ignores insider trading seems like it would also spark outrage, though this hasn’t happened to Polymarket yet (beyond it being illegal for Americans to participate in for other regulatory reasons).

Elicitation platforms, as well as play-money prediction markets, don’t have these problem. Nearly any subject is welcome, opening up questions about elections, war, and (in the case of Manifold) even users’ personal lives. Their problem, from a PR perspective, is that they often come off as nerdy hobbies.

6. Liquidity

Prediction markets need liquidity: some critical mass of contracts available on either side of the current market price so that a trader can buy/sell their desired amount without drastically moving the price. The most liquid and active questions on prediction markets tend to either be short term (sometimes even daily, in the case of Kalshi’s markets) and/or those about an interesting subject (such as the Presidential election or high-profile awards like the Oscars). A very meaningful market might not attract many traders, especially if it is a question that might take a year or more to resolve.

Elicitation platforms are much more versatile, because they don’t formally require liquidity (they merely get more accurate as more individuals participate, assuming no dropoff in participants’ quality) and participants aren’t forgoing any time value by predicting on a question (it should be less time consuming to forecast on one “what will be the unemployment rate at the end of December” question during the year, than on monthly questions about the unemployment rate). So a fairly small number of forecasters probably gets you a decent forecast, and there’s no hesitation to participate on a question that doesn’t resolve for years [3].

For prediction markets, one path to creating liquidity where it doesn’t naturally emerge is for the platform (or whoever creates a question) to subsidize it by seeding some initial liquidity to make a market around their initial estimate.  

7. Proprietary information

If you’re trying to recruit forecasters so you can contract with them as consultants helping you “predict the future” on proprietary questions, you’re better off recruiting on elicitation platforms.

First, because of position sizing, it’s possible to earn top-tier profits on a prediction market by sticking to a limited domain. Maybe a trader is skilled at modeling weather, or is a native Russian speaker, or even has inside information; maybe someone looking for a good forecaster wants these particular skills, but they aren’t generalizable advantages.

Second, prediction markets’ largest rewards often go to people who are fastest to see breaking news and trade it in the market. There are some cases where a rapid adjustment in the market’s probability is valuable (PredictIt is a massive improvement on live television coverage of election returns as “too close to call”, for example), and there’s certainly some transferable skills from figuring how to calibrate the right update to a prediction market question based on the exact news that’s been released, but it’s a different skill from producing accurate forecasts far enough in advance that they can shape decision making, which is often what is wanted from a forecast.

If you have enough participants that you can set up an internal prediction market among those with proprietary information and it will have (or you will provide) sufficient liquidity, that’s probably fine, but I believe it’s had mixed results in companies who have tried it.

8. Differences I want to think about more

• Handling questions with unclear or disputed resolutions
• Creating conditional questions
• Psychological differences between participants
• Which disadvantages can be fixed by innovations (for example, will many more participants trade long-term prediction market questions if the administrator of the market invests contract balances in money-market funds)
• Prediction markets as better incentives for original investigation and research

[1] Interested readers can get an introduction here: https://www.metaculus.com/help/scores-faq/

[2] https://en.wikipedia.org/wiki/Policy_Analysis_Market

[3] Metaculus launched a question “Will SpaceX land people on Mars before 2030?” in October 2016. Over four thousand users have entered a forecast.