Reported comparisons
Token estimates use
payload_bytes / 4, so the estimate may be fractional. This is a response-size approximation, not a model-specific token bill. The edgartools wire-byte measurement includes direct SEC retrieval; its zero-key, client-side architecture is a different design choice, not a defect.
Method
The benchmark uses authenticated production endpoints for hosted APIs and direct SEC EDGAR access for edgartools. It records wall-clock time from request start to result, including client-side parsing for edgartools. Each operation runs at least five times; p50, p95, and p99 use Hyndman-Fan Type 7 interpolation. The production workflow runs on GitHub Actionsubuntu-latest; within a workflow run the comparands execute from the same runner, but GitHub assigns runners from a variable pool and the comparand steps run sequentially.
For edgartools, the persistent cache is cleared between iterations to measure cold fetches. Payload measurement is raw hosted response bytes or the sum of direct SEC HTTP bytes for edgartools. Network conditions, API versions, cache state, rate limits, source availability, request shape, and client behavior all affect a real integration.
To reproduce a row, use the named operation and comparand, preserve the capture conditions, retain the raw observations and failures, then recompute the stated percentile from that set. Check response equivalence before comparing elapsed time: a result that lacks required fields, provenance, or the requested representation should be reported as a different outcome, not treated as a like-for-like win. A rerun is a new dated measurement, even when its setup matches this one.