A teardown of how Mercor builds, checks, launches and sells its APEX benchmarks, read from its site, papers and repositories.
The public APEX-Agents set is 9% of the licensable dataset. The benchmark is marketing; data licensing is the business. Verified
Benchmark pages in four sections. How much a page shows depends on how much of the benchmark Mercor owns. Verified
Average drop for three frontier models from public Terminal-Bench 2.1 to Mercor's held-out version. Verified
AI judge agreement with experts on APEX-Agents, from 747 labels. Quality claims come with numbers. Verified
Share of APEX-Agents tasks re-solved from scratch by experts who never saw them. 10% needed fixes. Verified
arXiv goes up 1 to 2 days before launch. Blog, board and data go live together. Agent-sourced
APEX-Accounting's task count on its page against its paper. Mercor's own numbers disagree in four places. Verified
Benchmarks sit under "Research", next to an expert marketplace, an enterprise arm and a program that buys companies' workplace data.
Benchmarks, leaderboards, methodology, off-the-shelf data.
Custom agents in production in 4 to 6 weeks. "Evaluate … with APEX".
Experts find paid work on work.mercor.com.
Pays firms for Slack, Gmail, Salesforce and GitHub data, then licenses it to labs.
Most of the catalog is other people's benchmarks run by Mercor. Only 8 pages are Mercor's own or a partner's, and those are the only ones with the full treatment. Verified
| What the page shows | Own + partner (8) | Extended (10) | Open-source (25) |
|---|
Ten "Mercor-extended" pages put each model's score on a public benchmark next to its score on Mercor's held-out tasks. The drop sells fresh data without saying so. Verified
The methodology page has 31 benchmark cards. Every card fills the same 11 fields. One rule applies everywhere: maximum reasoning effort, with repeats averaged into pass@1, never best-of. Verified
The public set is licensed for evaluation only and gated on Hugging Face. Training rights to the full dataset are what Mercor sells. No price appears anywhere. Verified
mercor.com/data asks firms to connect Slack, Gmail, Salesforce, GitHub, Drive, Notion and 100+ other tools. The data is de-identified automatically and licensed to labs. Payment comes 2 to 4 weeks after signing, with referral bonuses. Target industries: legal, healthcare, finance, professional services, manufacturing, retail, real estate, logistics.
Every partner card says "In partnership with [logo]". Behind it, who built the benchmark varies a lot. Partners are AI companies serving one industry each, plus Google.
Mercor publishes no checklist. Its papers describe the same sequence every time, and most steps come with a number. The three with the strongest numbers are outlined.
Also from the papers: PLE paid about $4,663 per expert-written task and $3,157 per co-written task. Benchmarks used 42 to 256 experts each. Top scores at launch: Agents 24%, SWE 25%, PLE 48%, Accounting 56%. Agent-sourced
Six launches across 14 months. The core package usually lands within a week. Open data sometimes trails by six to ten weeks. Agent-sourced dates
Versioning. A new version gets a new Hugging Face repo, and the blog post doubles as the changelog. The leaderboard URL is reused and old scores are replaced. APEX-Agents v1.1 removed scores for models caught "scattergunning". There are no changelog files, no pointers from old versions to new, and no DOIs.
Partner launches mostly go out on the partner's own channels first. Mercor adds a hosted board weeks or months later. Inferred from archive captures
Eight visible problems. Most come from keeping the same fact in several places.
APEX-Accounting is "Released March 2026" on the methodology page. Its paper and blog are from July 2026.
APEX-Accounting: 100 tasks on its page, a 160-task private set in its paper. Verified
"5M+ domain experts" on the data page, "4M+ experts" on the partner page. Verified
APEX-Accounting sits under partners, but the methodology page credits Mercor alone. There is no origin field.
35 of 43 pages have no uncertainty, no FAQ and no date.
Old repos stay up with no pointer to the new version. No changelog files.
The licensing catalog shows no per-dataset quality numbers. The rigor stays in the papers.
Judges are mostly frontier models from labs that are also on the boards. Mercor discloses the risk.
| Unknown | Who could tell us |
|---|---|
| Who signs off a release; any review board or partner veto | Former Mercor research staff; the APEX PM role (closed) |
| Reviewers per task, reviewer scoring, offboarding thresholds | Former Mercor contractors |
| Partner contracts: payment, ownership, co-publication rights | Staff at Ramp, Harvey, Cognition, Google Cloud AI Research |
| Licensing terms, exclusivity, pricing | Data buyers at AI labs |
| Difficulty threshold and discard rates | Not published |
| Revenue split across benchmarks, licensing, enterprise and data buying | Not public |