How we build it

01Architecture and MVP stack

The MVP (minimum viable product, the smallest version that works) is deliberately boring: one GitHub repo, scheduled GitHub Actions, flat JSON files, and a static site. No servers, no database. TS-Arena and ForecastBench both run on pipelines this simple, and it keeps every artifact public and auditable by construction.

The repo is live: github.com/jajamoa/social-sim-arena. This is the actual layout:

social-sim-arena/
├── questions/season0.json      frozen round definitions (id, lock_at, release_at, resolve rule)
├── ssa/
│   ├── adapters/votehub.py     poll-level JSON, no key (approval + generic ballot)
│   ├── adapters/fredcsv.py     Michigan sentiment via keyless FRED CSV
│   ├── average.py              house-effect-adjusted, recency-weighted average
│   ├── baselines.py            persistence + clamped straight-line trend (OLS)
│   ├── scoring.py              CRPS / RPS / skill (unit-tested)
│   └── refresh.py              orchestrator: fetch, compute, write site/data.json
├── forecasts/<round>/<entrant>.json   submissions, via pull request
├── entrants/<entrant_id>.json  agent registry (register by PR, schema-checked)
├── resolutions/resolved.json   ground truth as rounds resolve
├── schema/forecast.schema.json submission contract
├── tools/validate_submission.py       run locally and by CI
├── tests/test_scoring.py       hand-checked scoring values
├── site/                       entry page + data.json (generated)
└── .github/workflows/          refresh.yml (daily cron), validate.yml (PR check)

1. Cron: fetch. refresh.yml runs every 6 hours: pulls VoteHub and FRED, recomputes averages and live baselines, commits site/data.json. The entry page reads that file, so the site is always as fresh as the last cron run.
2. PR check: lock. validate.yml runs on every pull request touching forecasts/: schema check, round exists, and now < lock_at. A submission after the lock fails CI. Commit time is the timestamp; the sha256 of the canonical JSON is the fingerprint.
3. Resolve + score. When a release drops, the resolved value lands in resolutions/resolved.json (manual for now, cron later); the next refresh scores every entrant with CRPS and skill and publishes the leaderboard in data.json.

Everything is a commit. The audit trail is the git history itself, which is the cheapest possible pre-registration.

02Data sources and APIs

Ranked by how well each source fits a lock-before-release protocol. Gallup is gone: it ended its 88-year presidential approval series in February 2026, so nothing here depends on it.

GET https://api.votehub.com/polls?poll_type=approval
GET https://api.votehub.com/polls?poll_type=generic-ballot

GET https://fred.stlouisfed.org/graph/fredgraph.csv?id=UMCSENT

03Questions and submissions

Register first: one PR adding entrants/<entrant_id>.json (id, name, type, method; see schema/entrant.schema.json). The arena site has one-click prefilled GitHub links for both registration and forecasts.

A round is one question about one scheduled release. A submission is a distribution over the answer, filed before the lock. Submitting = opening a pull request that adds one JSON file; validate.yml enforces the deadline and the schema (schema/forecast.schema.json), and prints the canonical sha256. Validate locally first:

python tools/validate_submission.py forecasts/<round_id>/<entrant>.json

// questions/yougov-2026-w34-approval.json
{
  "round_id":  "yougov-2026-w34-approval",
  "tracker":   "economist_yougov",
  "question":  "Trump job approval, % approve among US adults",
  "release_at":"2026-08-18T14:00:00Z (est.)",
  "lock_at":   "2026-08-16T14:00:00Z",
  "resolve":   "topline PDF; crosstabs scored separately",
  "crosstabs": ["party_id", "age_4way", "race_4way"]
}

// forecasts/yougov-2026-w34-approval/gpt-5.5_web.json
{
  "entrant":  "gpt-5.5_web",
  "topline":  { "mean": 41.2, "sd": 1.4 },
  "crosstabs": {
    "party_id": { "dem": {"mean": 8.5,  "sd": 1.5},
                  "ind": {"mean": 35.0, "sd": 2.0},
                  "rep": {"mean": 84.0, "sd": 2.0} }
  },
  "notes":    "harness v0.3, temperature 0, 3 samples, median"
}

Point guesses are not accepted. Two formats, both scored with CRPS so they compete fairly: a normal (mean + sd), or a quantile set for skew and fat tails, e.g. "quantiles": {"0.05": 37.0, "0.25": 39.2, "0.5": 40.1, "0.75": 41.0, "0.95": 42.8} (must include the median). The canonical JSON is hashed (sha256) and the hash plus commit timestamp is the citation.

04Scoring spec

• CRPS for continuous toplines. Think of it as absolute error, generalized to distributions: confident and right scores best, confident and wrong scores worst.
• RPS (ranked probability score, the same idea as CRPS for ordered categories) for ordered crosstab bins such as age bands or agree-to-disagree scales.
• Skill is the headline number: 1 - CRPS(entrant) / CRPS(persistence), averaged over resolved rounds. Positive = better than copying the last release. Weather forecasters make the same move, scoring against 'same as yesterday' (persistence) and the seasonal average (climatology).
• Elo (a chess-style rating built from head-to-head results): computed pairwise per round, with confidence intervals from bootstrapping (resampling the rounds), following TS-Arena. Handles entrants joining mid-season.
• Resolution: an average across pollsters that corrects each one's habitual lean (its house effect), never a single poll. A single poll misses by about 3.5 points on average (RMSE, a standard error measure) per poll (Shirani-Mehr et al.), and 2020 pre-election polls missed by 4.5 points on average nationally (AAPOR), so single-poll resolution would mostly measure noise.
• Crowd: every round also scores the equal-weight mixture of all submissions as one distribution (the community aggregate). In the backtest the crowd of four basic models already beats each of them; whether any single agent beats the crowd is a headline question.
• Backtest: the same protocol replayed over 300+ real historical releases (Michigan sentiment since 2015, poll waves since 2025) fills the leaderboard before live rounds resolve, and validates the scoring stack end to end.
• Event rounds (policy launch, scandal, debate) are tagged, because persistence is hardest to beat in calm weeks and forecasting skill shows up around events.

05Model harness and human baseline

Every LLM runs in the same harness: same prompt template, temperature 0, three samples, median taken, full logs committed. Two arms (versions) per model, the same setup LLM-SoccerArena used: one closed book, one with web access. SoccerArena found web access improved the Brier score (an accuracy score for probability forecasts; lower is better) by only 0.023, and whether that holds for opinion is itself a result.

Run your agent continuously

An agent can live entirely on its own schedule: read the open rounds and live data from two public JSON files, produce forecasts, open a pull request. Copy templates/agent-cron.yml into your own repo and replace one step with your model; it then submits automatically every day until season end. The arena's own baselines run exactly this way: the refresh cron re-files their forecasts every 6 hours while a round is open, and the last commit before lock counts.

The human baseline is a paid panel of 5 to 10 forecasters on a question subset. This is not optional decoration: at Metaculus, pro forecasters beat the bot teams every quarter through Q2 2025, and ForecastBench's published leaderboards keep superforecasters ahead of the best LLM. A leaderboard without humans has no credibility anchor.

06Guardrails

• Aggregate only. The arena publishes population-level forecasts and scores. No individual-level targeting data exists anywhere in the pipeline.
• No microtargeting. Terms of use forbid using arena outputs to target ads or messages at specific groups. Precedent: EU Regulation 2024/900 and OpenAI's ban on demographic-targeted campaign material.
• Quiet window. No new election-linked rounds open in the final 72 hours before election day. Locked forecasts stay sealed until polls close. EU countries impose quiet periods of 1 to 15 days; we adopt the spirit.
• Labeling. Synthetic outputs are always labeled as model forecasts, never presented as polls. This follows the 2025 ICC/ESOMAR code revision and AAPOR's 2026 AI task-force direction.

07Next steps

Toward a solid paper, before the ICLR deadline

1. Real model runs. Wire the six providers into the harness (keys via Actions secrets), replace every placeholder with real output, and re-run the full 2015-2026 backtest through the same harness: fixed prompts, temperature 0, three samples, logs committed.
2. A bigger question set. Add state-level approval, per-wave crosstab targets (party, age, race), and issue questions. Run a power analysis: how many rounds are needed before model differences clear the bootstrap confidence intervals.
3. Scoring rigor. Sensitivity analysis of the house-effect parameters (window, halflife, shrinkage); robustness of rankings under alternative resolutions (single poll, unadjusted average); calibration reporting (PIT histograms, interval coverage) alongside CRPS.
4. Stronger baselines. AR(1) and a local-level Kalman filter, Holt-Winters, a fine-tuned small model, and a paid human forecaster panel on a question subset. The human anchor is what makes the leaderboard credible.
5. Contamination audit. Verify release dates against each model's training cutoff, plant canary questions, and document lock enforcement with commit-hash evidence in an appendix.
6. Pre-registration. Freeze season 0 questions and the scoring spec in a public pre-registration; print the midterm forecast hashes in the submitted paper.
7. Analysis with a finding. Skill decomposition into calm weeks vs event weeks; why pooling wins (error correlation between models); whether any single model beats the crowd; per-target difficulty profiles.
8. Writing. ForecastBench-style structure: protocol as the contribution, initial cohort results, living leaderboard. Related-work table against OpinionQA, SubPOP, and the retrospective silicon-sampling literature.

Engineering, in value order

1. First-party adapters. Economist/YouGov crosstab PDF parser and Morning Consult HTML parser kill the five-week VoteHub lag; archive every raw fetch under resolutions/raw/ and pin adapter fixtures as regression tests.
2. Resolution automation. Release detectors that open an auto-resolve pull request with the official value, a screenshot, and a hash; the scorer excludes any forecast whose first commit on main postdates the lock.
3. Harness productionization. Provider implementations with retries and cost caps, per-round logs committed, weekly spend report.
4. Human pipeline. A cron that parses the guess issues into one pooled human-crowd distribution per round.
5. Site. Official logo SVGs in site/logos/, a per-round detail page with the forecast strip and resolution replay, a share card per resolved round, and a Chinese mirror page.
6. Operations. Freshness alarms (data older than 45 days fails the cron and opens an issue), value sanity bounds, a backtest determinism test in CI, pinned dependencies.
7. Data quality. Dual-source cross-checks with divergence alerts, and the quarterly commissioned never-asked questions as the contamination-free gold standard.

08References

• ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities · ICLR 2025. Dynamic contamination-free questions, superforecaster baseline.
• TS-Arena · ICLR 2026. Live time-series arena; forecasts registered before the data exists; Elo with bootstrapped confidence intervals.
• LLM-SoccerArena · Jul 2026. Prospective live protocol, factorial design, 2026 World Cup.
• WorldCup Arena · Aug 2026. Six frontier LLMs tie the bookmaker-favorite baseline.
• Metaculus AI Benchmark, Q2 2025 results · Pros beat bots every quarter through Q2 2025.
• When Synthetic Users Fail · Jul 2026. LLMs lose to a demographics-only baseline and homogenize subgroups.
• Bisbee et al., The Perils of LLMs · Political Analysis 2024.
• Argyle et al., Out of One, Many · Political Analysis 2023. The optimistic origin of silicon sampling.
• "AI polls" are fake polls · Silver Bulletin, Apr 2026.
• AAPOR 2020 pre-election polling report · 4.5 pt average national miss.
• Shirani-Mehr et al., Disentangling Bias and Variance in Election Polls · JASA 2018.
• Jackman, Pooling the polls · 2005. House effects and latent-state averaging.

09How to join

• Adapters: pick a source above, write the fetcher and parser, add a fixture test.
• Harness: prompt template, API runners, retry and logging.
• Scoring: CRPS / RPS / Elo implementations with unit tests against hand-computed examples.
• Backtest: replay 2015-2025 tracker archives through the full pipeline.
• Human baseline: recruit and coordinate the forecaster panel.

Open tasks, in priority order

• Fresher adapters: Economist/YouGov crosstab PDF parser, Morning Consult HTML parser, Civiqs reader, Silver Bulletin CSV. Each one upgrades a "manual" or "lagged" source to machine-fresh.
• Resolution cron: auto-detect releases and write resolutions/resolved.json instead of manual entry.
• LLM harness: uniform prompt runner over 10+ models, closed and web arms, logs committed.
• Backtest: replay 2015-2025 archives through the scoring stack for the paper's validation section.
• Elo with bootstrapped CIs on top of the per-round CRPS records.

Ping Chance. Code and issues: github.com/jajamoa/social-sim-arena.