Skip to main content

​
Omni Datastream 2.0 update

This page tracks the final status of the 2.0 execution lane: what shipped, what was proven, and what remains as follow-on cleanup rather than core 2.0 blocker work.

​
Final shipped scope

  • live Mintlify docs on docs.secapi.ai
  • compact and full section response modes across REST, MCP, CLI, and SDK flows
  • hosted MCP install metadata and publish-ready JavaScript SDK
  • benchmark, freshness, coverage, and migration documentation
  • section extraction stage profiling and autoresearch control wiring
  • historical ownership and insider backfill expansion:
    • EDGAR-derived universe discovery
    • filing-level issuer resolution for insider forms
    • truthful provenance reporting in backfill summaries
  • fast-follow fixes for:
    • statement alias normalization
    • ticker/cache normalization
    • replay retry and DB pool hardening
    • PR 23 and PR 28 review findings
  • manifest-first section extraction on the hot path
  • persisted section manifests/snippets on startup and cold misses
  • Rust extractor prototype and benchmark harness
  • final live acceptance benchmark proving the section path now beats sec-api.io

​
Final acceptance result

The last material competitive gap for 2.0 was the live section-extraction path. That gap is now closed. Key result from run 23166765587:
  • section_item_1a_aapl: Omni 155.36ms p50 vs sec-api 295.79ms p50
The supporting closeout read is now:
  • Omni leads on mapping_lookup
  • Omni leads on filing_search
  • Omni leads on xbrl_to_json
  • Omni leads on the benchmarked financialdatasets.ai structured statement and filings paths
  • Omni now also leads on the final live section-acceptance path used to close OMNI-500

​
Production decision that closed the lane

The production winner is:
  • TS/Bun manifest-first extraction
The Rust lane remains:
  • a prototype and benchmark path
  • useful for malformed HTML and future parser work
  • not the production hot path today
Why:
  • the measured recommendation after the Rust benchmark lane was keep_ts_manifest_first
  • the measured parser-stack recommendation was stay_current_stack_with_opportunistic_anchor_indexing
  • the current production path already clears the live acceptance benchmark without incurring Rust cutover complexity

​
Operator notes

  • same-space extraction autoresearch is intentionally paused
  • it does not need to reopen right now because the live section gap is closed
  • if reopened later, it should stay limited to the new manifest-backed search space
  • benchmark, freshness, and coverage claims should continue to come only from tracked source-of-truth artifacts

​
Remaining non-core follow-up work

  • refresh the public docs pages so all benchmark/status pages reflect the final 2.0 closeout
  • simplify secapi.ai route linking so canonical docs and changelog live under docs.secapi.ai
  • do the omni-apps post-cutover hygiene pass to remove dead legacy SEC fallback paths
  • optionally certify endpoint-by-endpoint 5-year historical support if a broader public coverage claim is needed