FAQ

FAQ

Questions people ask about the arena. New here? Read the background story. Entering a simulator? Read the docs. To add a question, open a pull request against this page.

Static benchmarks

Eval-via-forecasting seems like a solid reality check. Could it expose biases we miss in standard benchmarks?

Good question! I think this is exactly where forecasting can add something beyond standard benchmarks.

First, it's live: forecasts are locked before the real outcome even exists, so models/agents can't benefit from contamination or memorization. The questions and scoring rules are pre-registered, and every forecast and result is public.

Second, the outcome is generated independently of us and the model builders, through existing real-world surveys and trackers. We don't get to decide what the "right answer" should look like. Instead, the world resolves it for us.

And importantly, we don't only look at the overall average. We also evaluate demographic breakdowns, so a simulator can't look good just by getting the topline right while systematically misrepresenting particular groups.

One-liner: Does what we learn from standard benchmarks actually hold up in the real future?

More on fair evaluation, anti-cheating, and responsible use in our upcoming technical report. The lock, the shared cutoff, and the public timestamp are also on the docs under Integrity.

Forecast arenas

Where is the difference from a forecast arena in the general sense?

The object of the benchmark is different.

Forecasting benches such as ForecastBench, FutureX, and Prophet Arena mainly test whether a system has general future-prediction ability, so many of their tasks are scattered, one-off events.

SSA watches a population that already exists and is measured again and again. The same American consumers' sentiment about the economy can be asked every week. The question itself may not change; the public does, as the world does. What we are really testing is whether a simulator has that population modeled.

Some Season 0 tasks therefore look like ordinary forecasting: predict the next observation. Population structure is already in the score — joint subgroup profiles, not only the national number. As sources expand, we will add tasks that press harder on population modeling:

  • New-question extrapolation. The first time a public is asked something with no history of its own — for example, how far people would authorize an AI agent to complete banking, shopping, and medical appointments for them — the forecast has to come from what is already known about that population.
  • Intervention / counterfactual. Tell a subset of people that a policy would cost an ordinary family $240 a year, then forecast how their attitudes move.
  • Subgroup prediction. Not only the overall number, but 18–29-year-old women, highly educated men, urban renters, first-time voters.
  • Cross-question / temporal consistency. The same population, different questions, different weeks — do the forecasts stay coherent?

These tasks test heterogeneity inside the population, and consistency across questions, times, and subgroups.

If the only goal is generic future prediction, the thing you train may be a very strong forecaster, or even a very strong researcher. That is a different object from a simulator that can keep interpreting, generating, and predicting how a public behaves.

On data, we use recurring public sources, look for partners who measure the same populations, and will keep collecting measurements of our own.

Something still unclear? Open an issue, or write to hi@social-atoms.org.