Filing Signal

A benchmark for primary-source financial document work, proposed to the Kaggle Benchmarks team by AITrailblazer. This page is the public record of the proposal.

Filing Signal grades AI systems on one judgment: whether a real SEC filing contains material signal. The system receives a complete filing packet, the same material a working analyst reads, and returns a verdict of SIGNAL, NO SIGNAL, or ABSTAIN, graded against a sealed answer key.

The filing judgment is not presented as a lone, narrow label. It is the first module of the wider buy-side analyst task family: reading primary documents, separating what matters from what is routine, and stating a defensible call. Investment memos, earnings reviews, and position decisions are all built from this judgment, made filing by filing. A benchmark that measures the judgment at its source measures the foundation the rest of the workflow stands on.

The problem: what the Kaggle audit found

In a talk posted on October 25, 2026, Janie Gu of Google DeepMind, who leads the Kaggle Benchmarks team, reported that her team spent two months reproducing 65 industry benchmarks, running each of them multiple times and reading the execution traces. The reproduction found failures in the benchmarks themselves, not only in the models:

• Credential leakage: an evaluated model found API keys for a different model inside the evaluation container, called that model, and submitted the response as its own.
• Grader tampering: evaluated systems modified the grading script to award themselves a score of 100 percent, because the evaluated system and the grader shared one container.
• Answer lookup: models searched public sites for answers instead of doing the work.
• Mismatched reasoning effort: harness defaults silently ran models at different reasoning effort levels, so published comparisons were not head to head.
• Rerun variance: the same benchmark, on the same model, the same harness, and the same infrastructure, produced scores that moved by up to plus or minus 20 percent between runs.

Source: Janie Gu and Suyeol Yun, "Why Frontier Models Fail as Buy-Side Analysts," AI Engineer, posted October 25, 2026: youtube.com/watch?v=aVbRSIQO2dE.

Construction, mapped to the findings

Filing Signal is built so that each failure the audit named is refused by construction, not by policy:

• Sealed answer key. The key is hash-committed before any run and is never hosted on Kaggle. Answer lookup has nothing public to find, and the live key is never exposed to the evaluated system.
• Isolated grading. The grader runs separately from the evaluated system. No other model's credentials exist inside the evaluation container, so there is nothing to steal and no grading script within the evaluated system's reach.
• Matched reasoning effort. Reasoning effort is set explicitly for every system under test, and the setting is printed with every published score.
• Repeated runs, variance printed. Every published score is the mean of repeated full runs, and the rerun variance is printed beside the score. A ranking inside the variance band is reported as a tie, not a result.
• Planted canary items. Every grading batch contains planted items whose correct behavior is known by construction. A canary that moves flags the batch for tampering or leakage before any score is published.
• A scored abstain band. A system may decline an item at a stated cost instead of guessing. A grader forced to answer everything fabricates; the abstain band makes calibrated refusal a measured skill.
• A renewing item pool. New SEC filings arrive continuously, so the item pool renews by construction and items postdate model training cutoffs. Memorization decays on its own.

The evidence record

Filing Signal is proposed by a practice that already grades in public. The record below is stated with its dates and measure titles, and nothing in it is rounded up.

• DeltaSignal Decider corpus (locked October 2026). 179 real SEC filings carry locked v3 final labels of 22 SIGNAL, 155 NO SIGNAL, and 2 ABSTAIN. The finals were built by two blind grading passes, a fresh-persona judge blind to label origin for disagreements, and a human lock. These are machine grading passes under a frozen rubric, settled by adjudication and locked by a person; they are not human hand labels.
• Track record (August 1, 2025 to October 13, 2026). 32 calls published in DeltaSignal articles were graded correct against the outcome, each with its original post on X linked as proof. The full record is public at atlas7.app/track-record, with one page per ticker at atlas7.app/tickers.
• Technical report (version 1.2, October 29, 2026). The full benchmark writeup: task design, answer-key construction, the two-program grader, the custom model, and all measured results against the locked key, with the grading infographic as an appendix. Public at atlas7.app/filing-signal-report.pdf.
• The Empty Unison (tally as of October 29, 2026). A public grading of machine-written crypto market summaries against the primary record: 60 entries graded, of which 37 CONFIRMED, 18 UNSUPPORTED, 3 SILENT-ON-REAL-DRIVER, and 2 NO CAUSE, with 30 revisions by the feed caught and documented. Public at atlas7.app/empty-unison.
• Skills Record. A graded public record of AI agent skills under a frozen standard, public at atlas7.app/skills.

How PAIEC fits

PAIEC-2026 is the Predictive AI Evaluation Competition run by the Stanford AIMS Lab as part of a NeurIPS 2026 workshop. Its task is a different one from Filing Signal, and the two are related by method, not by subject. In PAIEC, a submitted program predicts, for an AI system and a benchmark item, the probability that the system answers the item correctly, without running the system. Predictions are made under benchmark-level cold start, on benchmark families the program has never seen, with label budgets of 0, 1, 3, 7, 15, and 31 revealed labels per subject and benchmark pair. The score is Brier ALC, an average of Brier scores across those budgets.

Our entry's selected submission is v2, Codabench submission ID 1016554 in competition 17828. It finished the Codabench evaluation with a Brier ALC of 0.183081, the only finished Codabench score in this entry. Its method is a fitted two-parameter logistic item-response model with a metadata prior for unseen subjects, Platt calibration, and shrinkage toward the subject's own revealed mean.

The entry also built a v3 method, which changed how revealed labels are consumed. On two internal benchmark-holdout splits, v3 improved on v2. That result is internal validation only: every v3-family submission filed to Codabench failed on the organizers' backend, five at the scoring stage and one at the prediction stage, so v3 has no official score and none is claimed for it here.

The technical report was filed to the NeurIPS 2026 PAIEC workshop; the filing is on OpenReview at openreview.net/forum?id=Tcno4NY5YQ. The report itself is published here, under the CC BY 4.0 license it was filed under, at atlas7.app/paiec-2026-report.pdf, so it can be read without waiting for the venue's release.

The fit, stated plainly: PAIEC is the cold-start, calibration, and label-efficiency method layer. It is evidence that this practice can predict benchmark outcomes under a budget, calibrate probabilities, and report a failed line of work in full. Filing Signal is the real-workload benchmark. Neither is presented as proof of the other.

Relation to MandateBench

MandateBench, built by LinqAlpha with Google DeepMind and Kaggle, grades the full investment memo workflow: a fund mandate goes in, a long, short, or pass investment memo comes out, across a planned 50 tasks launching in November 2026. It scores the reasoning path behind the memo rather than a single golden answer, because two analysts can reach different calls and both can have done the work well.

Filing Signal is complementary, and upstream. A memo is built from filing-level judgments made one document at a time: this filing carries material signal, that one is routine. Filing Signal grades that judgment at its source, against a sealed key, on an item pool that renews as new filings arrive. MandateBench measures the finished analyst product; Filing Signal measures the perception the product is built from. The two benchmarks test different layers of the same task family and neither substitutes for the other.

The ask

In the October 25, 2026 talk, Janie Gu announced benchmark-builder support for Q4: a grants pool of $5 million, in her words "to fund any benchmark creation or research into fixing this space," together with free compute and frontier model inference for benchmark builders and a forward-deployed engineer from Google DeepMind to help co-design the evaluation. Filing Signal asks for consideration under that announced support.

Two facts are stated plainly. First, no public application form, deadline, eligibility rule, per-grant amount, or program terms were published with the announcement; the stated route was to reach out to the Kaggle Benchmarks team directly. Second, this page is the public record of the proposal. It is not a claim of an award, of eligibility, or of any existing arrangement with Kaggle or Google DeepMind.

Conditions

• The answer key and the live items never reside on Kaggle. Method and results are fully public; the data stays sealed.
• The canonical record of the benchmark, its runs, and its published scores stays on atlas7.app.
• Any co-design begins only under a written agreement covering the intellectual property in co-built material and the right to publish results.
• No rated party pays for ratings, in either direction: the parties graded by this practice do not pay for their grades, and this practice does not pay for placement.