Agentic Forecasting Atlas

Every experiment in this repo — a two-line naive baseline on gasoline CPI, a curriculum-trained oil analyst — walks the same loop. Data enters through adapters, one fetch() per source; there is nothing special about the built-in StatCan, FRED, and yfinance adapters, and nothing stops you from wrapping a CSV, an API, or a database the same way. Registered on a DataService, the data is fenced behind a cutoff, handed to predictors as one fresh context per forecast origin, and comes out the other side as scored, cached, comparable results.

That scored loop is Track 1. The same agents can also be used interactively — scenario analysis, monitoring, open-ended Q&A — without emitting a Prediction the harness can score (Track 2). This atlas is about the scored loop.

The two layers matter: the core library aieng.forecasting owns everything drawn solid below. You author the dashed pieces — a dataset, a spec, a predictor lineup — and the loop does the rest.

A backtest is only honest if every forecast is made from what was knowable at its origin — and "knowable" is not the same as "existed". A market close exists the moment the bell rings but publishes the next morning; official statistics trail their reference month by weeks. The repo encodes this as data, not discipline: every row carries a released_at stamp alongside its timestamp (absent the stamp, the timestamp stands in), and predictors can only see through the fence it defines. The same fence covers context — news snapshots and documents are cutoff-scoped exactly like series rows.

Everything that forecasts implements the same two-member contract: a predictor_id and a predict(task, context) that returns one prediction per horizon. That is the whole trick that makes the leaderboard possible — a naive baseline, a gradient-boosted model, a single structured LLM call, and a multi-turn agent all answer the same question from the same fenced context, so their scores mean the same thing.

The four families below are less a taxonomy than an escalation of how much machinery sits between the context and the answer. Keep the left two honest and strong: an agent is only as interesting as its margin over the best baseline you can field.

Specs are experiment design, kept in YAML and out of code: a task, a window of forecast origins, a stride, a warmup — or, for irregular calendars like BoC's meeting dates, an explicit origin_dates list. Because the lineup and the spec are independent, the same predictors run against a two-origin smoke spec, a full development backtest, and a protected evaluation — without edits.

The two loops answer different questions. backtest() is the open loop you run freely while developing and tuning. evaluate() is for the answer you commit to — and tuning against it destroys the answer. Attach an EvalTracker and a max_runs cap and the harness refuses extra runs and never caches, so spend stays visible. Call it without a tracker and the budget does not apply.

Identity (what the agent is) and role (its job in one experiment) are deliberately separate objects. The identity — persona, toolbelt, skills, model — is an AgentConfig; the role — what payload it sees per origin and what structured forecast it must return — lives on the AgentPredictor that wraps it. One identity can play many tasks, and rival strategies can play the same task fairly.

The capabilities column is where the fence from §02 gets stress-tested: the search tool runs a grounded sub-agent whose brief passes through an independent verifier before the analyst ever sees it. It reduces leakage; it does not eliminate it.

Same loop, same interface, same fence — different data, payloads, and machinery switched on. The matrix reads horizontally as "what does this implementation exercise" and vertically as "where do I find a working example of this component".

The numbering is not arbitrary: it mirrors the bootcamp progression — conventional methods, then LLM processes, then agents, then agentic evaluation. Each step assumes the last and adds one idea.

The architecture exists so that a new forecaster is always one of two paths — extend a reference implementation, or bring your own dataset — and every move on either path has a step-by-step guide in the repo.