Discovery, backtested

A staged pipeline of language-model agents

Sibyl runs as a staged pipeline of language-model agents, each with a narrow job. A high-volume triage agent (Claude Haiku) scores paper relevance; full-text reasoning agents (Claude Sonnet) extract structured, quote-anchored claims, compile them into a knowledge base, and generate predictions under explicit falsification conditions. Two stages — knowledge-base compilation and prediction generation — are gated behind mandatory human review, so the system is autonomous by default but never runs unsupervised at the points where an error would compound. Every extracted claim carries a verbatim source quote and a bibcode, so a finished prediction traces end-to-end back to the printed literature rather than to the model's parametric memory.

Temporal backtesting

The pipeline is evaluated the way a trading strategy is: by backtesting. The corpus is split at a fixed 2014 cutoff; predictions are generated from pre-cutoff literature alone and scored against post-2015 publications, counting a prediction as confirmed only when the later result matches its predicted direction and the confirming data was genuinely unavailable beforehand. The cutoff places three field-changing programs — LIGO's first observing run, NICER, and eROSITA — entirely inside the validation window. Of 60 predictions generated from pre-2015 work, 11 were confirmed by independent later results, spanning four X-ray-binary families rather than a single well-studied system; the confirmed rate holds between 12.5% and 18% across increasingly conservative provenance filters.

Provenance and honest failure

What separates Sibyl from a pipeline that simply emits plausible text is that it audits itself. A post-hoc provenance audit checks every extracted claim against its cited paper — quote, source class, formula, and sample size — and a cross-prediction consistency check catches fabricated content that per-claim quote checks miss. That audit surfaced four failure modes, ordered by how fixable they are: schema underspecification, corpus contamination, validation-era leakage, and citation hallucination. Reporting them, rather than burying them, is the point — the confirmation rate above is quoted under the filters that remove each one.

Key references: Sibyl posters at the ICML 2026 AI4Science workshop — Temporal Backtesting for Literature-Based Scientific Discovery with LLM Agents and A Multi-Agent Pipeline for Autonomous Hypothesis Generation.