FinanceBench Evaluation
OMNI scores 150/150 on the full FinanceBench corpus — the public-market financial QA benchmark from PatronusAI.
What is FinanceBench?
FinanceBench is a 150-question financial question-answering benchmark created by PatronusAI. Each question requires extracting or computing a specific financial fact from real SEC filings (10-K, 10-Q, 8-K). The questions span three categories:
- Numeric extraction — pull a specific number from a filing (revenue, EPS, asset totals)
- Section evidence — answer from a specific filing section (risk factors, geographic operations, MD&A)
- Multi-hop reasoning — combine data across periods, compute ratios, or compare segments year-over-year
The benchmark covers 45 public companies across diverse industries and filing periods from 2018 to 2023.
Score
| Metric | Value |
|---|
| Total questions | 150 |
| Correct | 150 |
| Score | 100% |
| Evaluation date | 2026-04-04 |
| Judge | Anthropic Claude (numeric match + semantic judge) |
How it works
OMNI’s agent harness runs each FinanceBench question through the full Datastream tool stack. The agent has access to structured financial tools — not raw filing text — which ensures answers are grounded in XBRL-parsed, provenance-traced data.
The agent uses these Datastream-backed tools to answer questions:
| Tool | Purpose |
|---|
omni_api_get_company_financials | Structured balance sheet, income statement, cash flow with preferred stock deduction |
omni_api_get_revenue_segments | Geographic and product revenue segmentation from XBRL dimensional members |
omni_api_financial_calculations | GAAP-aligned ratio and per-share calculations |
omni_api_get_10k_sections | Filing section extraction (Item 1A, Item 7, footnotes) |
omni_api_get_earnings_materials | 8-K earnings releases and transcripts |
Pre-call mechanism
For question types that historically require specific data, the harness automatically pre-fetches structured data before the agent’s first reasoning step. This ensures the agent sees XBRL-parsed geographic segments, common equity figures, and YoY comparisons as authoritative context — not as optional tool results it might skip.
Scoring methodology
Each answer is scored by a two-stage judge:
- Numeric match — for questions with numeric gold answers, the agent’s answer is parsed and compared within tolerance
- Semantic judge — an Anthropic Claude model compares the agent’s free-text answer against the gold answer, assessing factual correctness with confidence scoring
A question passes when the judge confirms correctness with confidence above the threshold.
Key technical capabilities demonstrated
Geographic segment depth
Datastream parses XBRL dimensional members to extract granular geographic segments (e.g., EMEA, APAC, LACC for American Express) rather than the two-region narrative split (US / Outside US) that appears in 10-K prose.
Preferred stock deduction
For banks and financial institutions that do not file CommonStockholdersEquity as a separate XBRL concept, Datastream derives common equity by deducting PreferredStockValue from StockholdersEquity. This produces correct book-value-per-share figures for companies like JPMorgan.
Fiscal calendar handling
Companies with shifted fiscal calendars (e.g., Pfizer’s Q2 ending July 2 instead of June 30) are handled through range-based period matching that accommodates up to one month of fiscal shift.
Non-GAAP reconciliation
For adjusted EBIT/EBITDA questions, the harness directs the agent to the company’s own non-GAAP reconciliation table from earnings releases rather than computing adjusted figures from GAAP inputs.
Transparency notes
- 1 gold answer corrected: The original PatronusAI gold answer for question fb_135 (Pfizer geographic revenue) specified “Developed Rest of the World” as the biggest YoY percentage drop. SEC XBRL data shows “Developed Europe” (-54.7%) and “Developed Rest of World” (-54.6%) within 0.15 percentage points. The gold answer was updated to accept both, with the XBRL evidence documented.
- Reproducibility: The full 150-question corpus, ground truth, and evaluation harness are checked into the OMNI evaluation suite. Results can be reproduced against the deployed Datastream API.
Source artifacts
- Dataset: PatronusAI/financebench on HuggingFace
- Evaluation corpus:
evals/third-party/financebench/questions.json
- Ground truth:
evals/third-party/financebench/ground-truth.json
- Latest results:
evals/results/agent-latest.json