What is measured
1
Named operations
The suite covers named operations such as entity resolution, filing search, section extraction, structured statements, and metrics. The exact operation and request shape determine what a result means.
2
Latency and response size
The artifacts record elapsed time and response-size estimates for the tested requests. They do not measure the usefulness or correctness of every possible response.
3
Bounded evaluation prompts
The repository also contains a separate investor-intelligence corpus and FinanceBench evaluation. Those are bounded test sets, not evidence that a single request answers every research question.
4
Reproducible context
Interpret a result with its capture date, comparand, operation, runner configuration, and methodology. Those details are part of the claim.
Reported comparisons
Hosted API comparisons
The comparison data was captured on
2026-03-18, and its scorecard was last regenerated on 2026-06-10. It reports p50 ranges for selected sec-api.io and financialdatasets.ai operations. Read the operation table and method before using a number externally.edgartools comparison
The
2026-04-23 edgartools comparison is preliminary. Its timings include direct SEC retrieval and client-side parsing, so it is not interchangeable with a hosted API measurement.FinanceBench evaluation
FinanceBench is a separate, bounded QA evaluation with its own question set, harness, and reported limitations.
Source and freshness
Benchmark speed does not establish that a result is complete, current, or suitable for a decision. Inspect route-specific source and freshness fields separately.
Reproduce a comparable workflow
- Choose one named operation and fix the issuer, filing or period, response view, filters, and expected evidence fields.
- Run each implementation from the same runner configuration, region, authentication posture, and capture window. Record versions, cache treatment, concurrency, retries, and failures.
- Keep the raw timing and response-size observations with the summary. State whether payload size is wire bytes, serialized body bytes, or another defined measure.
- Review response equivalence before comparing elapsed time. A smaller or faster response that omits required evidence is a different workload.
- Publish the capture date, sample count, summary statistic, and limitations next to any result. Re-run after material changes instead of carrying a dated result forward as live telemetry.
A responsible comparison
State the capture date, the named comparison, and the operation. For example: “In SEC API’s first-party 2026-03-18 benchmark, the reported p50 for the named operation was lower than the reported comparand p50.” Do not turn a dated, scoped measurement into a universal speed, payload, coverage, or extraction claim.Freshness and MCP comparisons
The benchmark artifacts are evidence, not live telemetry. When citing them, retain the capture date, methodology, and source artifact rather than presenting the numbers as current production behavior. For hosted MCP, check the live install metadata atGET /mcp before benchmarking or debugging an agent setup. The metadata exposes the deployed API’s current streamable HTTP transport, auth expectations, client install hints, and capability summary; use it as the install contract when local notes or cached docs disagree.
When comparing MCP and REST, compare the same workflow on the same day with the same auth, region, issuer, filing, and output mode. Keep requestId and traceparent with the result so support can separate tool latency, transport behavior, quota or circuit state, and source-data freshness. Describe client-side orchestration separately from API or tool time; an end-to-end agent turn is not interchangeable with a single REST request.