Skip to content
Menu

Look

Language

GitHub — @IchiSieben
Projects

Signal Agent

In development prototype

A market monitor where the model has to write a falsifiable prediction before it can be scored — 3 open, 0 resolved, the first ones due January 2027.

A market monitor that collects on a schedule in GitHub Actions and then makes a model account for itself: every judgement is signed, every call ships a falsifiable prediction with its resolution criteria written down in advance, and the accuracy ledger is the actual product.

1 min read

Technology

  • AI & agents
  • Automation

Sector

  • Economics & finance

Problem

Most agent demos are unfalsifiable. The model produces a confident paragraph, nobody writes down what would have counted as wrong, and next month there is a new confident paragraph. The interesting question is not whether a model can summarise a market — it is whether you can hold one to account afterwards, which means deciding in advance what would count as a miss.

What it will be

A case study on the part that is the point: the schema. A judgement records who made it (`judged_by`, one of claude-code, a human, or an import) and its verdict; confidence is forced to NULL when the verdict is no-change, so nothing can be confidently non-committal. A prediction carries a binary `claim`, the `resolution_criteria` written before the outcome is known, and a horizon that fixes the date it resolves. An outcome is correct, incorrect, or genuinely undetermined — and undetermined is excluded from the accuracy rate instead of being rounded into it. Re-judging never reads the previous verdict, so old confidence cannot anchor a new one. An indicator that fails to beat a coin flip over ten resolved predictions is flagged for removal. Collection and briefing run themselves; the judging step is run by hand inside Claude Code, and there is no LLM API key anywhere in the codebase.

Status: Phase 1 only: 95 observations, 6 signals and 3 open predictions collected. Nothing has resolved yet, so there is no accuracy number to show — the first predictions mature in January 2027, and the scoring commands are still stubs. Repo and demo follow.