AI competitions don’t produce useful models

The reason for open datasets and shared tasks where independent teams work on the same problem with the same evaluation is not to necessarily produce a production-ready deliverable. It’s to move the field forward. When shared tasks like this are run, the organizers normally require the participating teams to publish their work so that the community working on that class of problems not only has reference models, code, and data, but has a repeatable and comparable set of experimental results.

When teams work independently and don’t share their data or evaluation code, you inevitably run into issues of incomparable results, or worse invalid results due to accidents or even impropriety. Shared tasks are a proven way to make progress on difficult problems and provide valuable resources and results that might take significantly longer to reach if teams worked in isolation, or might never be reached at all.