Ryvx

Withdrawing our benchmark numbers

2026-08-11

On 2026-08-11 we published recall numbers for Ryvx against two benchmark corpora: 7/8 on DVWA and 6/8 on Juice Shop, each with a per-challenge list of what was found and what was missed. A few hours later we took the whole section down.

What we published

The claim was straightforward: run Ryvx against a known-vulnerable app with a fixed scoreboard, count how many challenges it solved, and publish that score next to which specific vulnerabilities it found and which it didn't. That's a reasonable thing to want from a security tool, and it's the kind of number that's easy to build a landing page around.

What the audit found

We went back and checked the two runs behind those numbers against their own artifacts — the audit trail, meta.json, the actual findings — rather than trusting the score that had already been computed. Neither run was what it looked like.

The Juice Shop run's root agent called finish while five subagents were still working. coordinator.shutdown() gave them a hardcoded 60-second grace period; they used all of it plus 24 more tool calls between them, and were then cancelled mid-turn. One of those subagents had a create_finding call rejected six seconds before the cutoff over a CVSS formatting error and never got the chance to refile it. The DVWA run didn't even get that far: it died partway through on an account-level API usage limit, and its own meta.json recorded root_completed: false.

Separately, and worse, the found/missed lists we'd published for each target didn't match what the runs had actually produced. DVWA was credited with finding stored XSS — it hadn't; that challenge was one of the misses. Juice Shop was credited with XXE, insecure deserialization, and SSRF findings that don't appear anywhere in that run's output. And at least one challenge was listed as missed that the scoreboard had actually recorded as solved. The list-building process, not just the scan, had failed.

Why the numbers were floors, not measurements

A truncated investigation's misses aren't real misses — they're questions nobody got to ask. A subagent cancelled mid-request didn't fail the challenge in front of it; it just never finished looking. Treating “6 of 8 solved” as a capability measurement assumes the other two were tried and failed, and for most of them that assumption was false. The honest description of 6/8 and 7/8 is a floor: at least this many, produced by an investigation that didn't get to finish.

What we fixed

The grace period that cut those five Juice Shop subagents off is no longer hardcoded. It's config.SUBAGENT_GRACE_S, defaulting to 600 seconds, overridable with RYVX_SUBAGENT_GRACE_S. meta.json now records cancelled_agents — how many subagents a run had to cut off — so a run that finished early can't be mistaken for one that ran to completion just by checking whether the root agent returned. scripts/benchmark_suite.py checks that field before it will score a run at all: anything with cancelled_agents greater than zero is marked void and excluded from solve-rate numbers, rather than scored and published anyway.

None of this is a claim that Ryvx now finds more. That isn't measured, and this whole post is about what happens when a number gets published before it's measured. What's fixed is narrower: a truncated run can no longer produce a score that looks like a finished one, and the tool that computes recall now refuses to compute it from a run it can detect was cut short.

The standing rule

Nothing replaces the withdrawn numbers until we run an untruncated scan against these corpora — one where cancelled_agents is zero, the cost cap wasn't hit mid-run, and the account's own API usage ceiling was checked ahead of time rather than found by dying on it. When that run happens, whatever gets published from it will be generated directly from that run's own artifacts — the same meta.json and findings the audit above was built from — rather than transcribed by hand into a table. That's the same failure mode twice, really: a human-assembled found/missed list is exactly the step that put the wrong challenges next to the wrong targets the first time.

We build a tool whose entire premise is “PoC or it didn't happen” — no finding gets reported without a working exploit behind it. Publishing a recall number we hadn't actually measured was the same mistake in a different place. Taking it down was the only response consistent with the rule we ask the tool itself to follow.