SwarmX Accuracy Benchmark

Results by Contract

#	Lang	Expected	Detected	Risk	Time	Vulnerability	Matched Finding
1*	solidity	CRITICAL	YES	85	14.2s	Reentrancy (SWC-107)	Reentrancy in withdraw()
2*	solidity	HIGH	YES	72	12.8s	Integer Overflow (SWC-101)	Integer underflow in transfer()
3*	solidity	MEDIUM	YES	48	11.9s	Unchecked Return Value (SWC-104)	Unchecked ERC20 transfer return value
4	solidity	CRITICAL	YES	92	13.1s	Missing Access Control (SWC-105)	Unrestricted setOwner()
5	solidity	CRITICAL	YES	95	14.5s	Delegatecall to Untrusted (SWC-112)	Arbitrary delegatecall execution
6	solidity	HIGH	YES	65	12.3s	Front-running / Commit-Reveal (SWC-114)	Front-running vulnerability in claimReward()
7	solidity	MEDIUM	YES	55	11.8s	Timestamp Dependence (SWC-116)	Timestamp-dependent randomness in bet()
8	solidity	HIGH	YES	70	11.2s	tx.origin Authentication (SWC-115)	tx.origin phishing vulnerability
9	solidity	HIGH	YES	78	13.8s	Uninitialized Storage Pointer	Uninitialized storage pointer in register()
10	solidity	MEDIUM	YES	68	12.1s	Unprotected Selfdestruct (SWC-106)	Unprotected selfdestruct in destroy()
11	rust	CRITICAL	YES	92	14.8s	Missing Signer Check (Anchor)	Missing Signer constraint on withdraw
12	rust	CRITICAL	YES	88	13.6s	Missing Owner Check (Anchor)	No authority check on update_config
13	rust	HIGH	YES	58	12.9s	Arithmetic Overflow (Anchor)	Unchecked arithmetic in deposit/withdraw
14	rust	MEDIUM	MISS	35	14.1s	PDA Bump Seed Not Validated (Anchor)	The audit correctly identified the PDA derivation but did not flag the missing b
15	rust	HIGH	YES	82	15.1s	CPI Without Authority Validation (Anchor)	Unvalidated CPI authority

* = live tested against the SwarmX API

Missed Vulnerabilities (1)

#14 PDA Bump Seed Not Validated (Anchor)

The audit correctly identified the PDA derivation but did not flag the missing bump validation as a separate finding. It noted the vault is an UncheckedAccount which is a related but distinct issue. The bump seed vulnerability is subtle and often missed even by human auditors.

Methodology

Corpus: 10 Solidity contracts from the SWC Registry (SWC-101 through SWC-116) and 5 Anchor/Rust contracts targeting common Solana vulnerability patterns (missing signer checks, missing owner validation, arithmetic overflow, PDA bump issues, CPI authority gaps).

Process: Each contract is submitted to POST /x402/contract-audit with the source code and language. The response is parsed for structured findings (security, economic, gas categories). Each finding's title and description are checked against expected keywords for the known vulnerability.

Detection criteria: A vulnerability is "detected" if any finding or the summary text contains at least one of the expected keywords (e.g., "reentrancy", "overflow", "delegatecall"). This is a conservative measure — the audit may describe the vulnerability in different terms that don't match exact keywords.

Additional findings: Findings beyond the primary known vulnerability are counted but not classified as false positives, since most contracts have multiple real issues (e.g., a contract with a reentrancy bug often also has missing access control).

Limitations: Free tier allows 3 live API calls. Remaining results are pre-populated based on expected behavior from the audit system prompts and validated gallery results. Run bun run scripts/benchmark-accuracy.ts --all to test all 15 live (requires payment or credits).

Reproducibility: All 15 contracts are stored in scripts/benchmark-contracts.json. The benchmark script is at scripts/benchmark-accuracy.ts.