The death and life of prediction markets at Google

Here’s a detailed post by someone who tried to make internal prediction markets successful at Google - for the second time. From the article:

Prophit worked, in part, because of internal transparency. When a market forecast how many Gmail users would join next quarter, it was based on a value that was visible to everyone at Google. Google was famously internally transparent compared to other tech giants.

One Waymo VP, upon seeing these safety metric forecasts, said this would help communicate these metrics across company divisions. And then, to my amazement, he said this was counter to his division’s goal to restrict information like this. The core mechanism of prediction markets — using the wisdom of crowds — can be antithetical to the common management desire to control who knows what.

…

In early February 2020, one month after I started my new role, I saw a question on Metaculus asking if a new coronavirus might lead to a global pandemic. One month later, all 150,000 Alphabet employees were sent home. Demand for information surged, and management didn’t have answers. So I used my extra time in lockdown to revamp and relaunch what I was now calling Gleangen as a prediction market for all Google employees.

Early markets, bet on by thousands of employees, centered on our uncertain future: when offices would re-open in various countries, the date vaccines would be available, and how at-home and in-office work would be balanced. Several times, such markets gave high probabilities to the creation of pandemic policies long before they were actually enacted.

…

I was well aware of the known obstacles to prediction market adoption in the workplace: unwillingness to share data, the desire for plausible deniability when projects fail, the risks of market manipulation, and good old-fashioned status quo bias. But in hindsight, my almost three-year struggle before I got official staffing for Gleangen came down to poor execution on exactly what Cowgill advised: understanding client needs. I had a somewhat utopian view of the value of information. I put some, but not enough, effort into figuring out the messy details on how to operationalize the wisdom of Googlers into a valuable resource for managers.

…

[T]o my dismay, I realized we hadn’t produced the information executives really needed. We asked questions of the type “Will Google integrate LLMs into Gmail by Spring 2023?” and “How many parameters will the next LaMDA model have?” Yet what executives would have wanted to know was “Will Microsoft integrate LLMs into Outlook by Spring 2023?” and “How many parameters will the next GPT model have?”

This turns out to be a general lesson from running a corporate prediction market. Forecasting internal progress, and acting on that information, requires solving complex operational problems and understanding the moral mazes that managers face. Forecasting competitors’ progress has almost none of these problems.