OpenAI Launches EVMbench: AI Benchmark for Smart Contract Security

OpenAI launched EVMbench with Paradigm to test AI agents on smart contract vulnerabilities. The benchmark evaluates detection, patching, and exploitation across 120 real audit flaws securing over $100 billion in crypto assets. Agents face economically critical blockchain environments.

EVMbench pulls from 40 Code4rena competitions and Tempo blockchain audits. Tasks run in isolated Anvil environments preventing live chain damage. Rust harness ensures reproducible deployments without unsafe RPC calls.

GPT-5.3-Codex scores 72.2% on exploit tasks versus GPT-5’s 31.9% six months prior. Detection and patching lag behind due to incomplete audits and functionality breaks. OpenAI released tools openly while committing $10M API credits for defensive research.

Three Core Evaluation Modes

EVMbench tests full security lifecycle capabilities.

Mode	Objective	Scoring Method	Difficulty
Detect	Full repository audit	Recall vs ground-truth + audit rewards	High
Patch	Fix vulnerabilities	Automated tests + exploit verification	Highest
Exploit	Drain deployed contracts	Transaction replay + on-chain simulation	Medium

Agents excel at explicit exploit goals but struggle with comprehensive audits. Patch mode fails most on subtle logic flaws without breaking core functionality.

Vulnerability Dataset

120 high-severity issues from real audits.

• 80% Code4rena competition findings
• 20% Tempo stablecoin payment contracts
• Severity range: Reentrancy to access control
• Asset protection: $100B+ open-source crypto

Tempo inclusion tests payment-focused code where stablecoin agents expected to scale.

Performance Results

Frontier models show clear task specialization.

GPT-5.3-Codex: 72.2% Exploit / 41% Detect / 28% Patch
GPT-5: 31.9% Exploit / 22% Detect / 15% Patch

Exploit success comes from iterative fund-draining. Detection stops after single finds. Patching breaks invariants frequently.

Technical Implementation

Rust-based evaluation harness provides:

• Deterministic contract deployment
• Restricted RPC eliminating live chain risk
• Transaction replay verification
• On-chain state simulation

Local Anvil fork prevents testnet pollution. All tasks fully sandboxed.

AI Behavioral Insights

Agents demonstrate distinct failure patterns.

Detection weaknesses:

• Single vulnerability fixation
• Incomplete repository traversal
• False positives beyond human baseline

Patch challenges:

• Subtle flaw removal breaks invariants
• Gas optimization conflicts with fixes
• Missing edge case coverage

Exploit strengths:

• Clear success metric drives iteration
• Transaction crafting proficiency
• Sandbox optimization

OpenAI notes benchmark understates real-world complexity. Grading misses novel findings.

Ecosystem Impact

$10M Cybersecurity Grant Program targets:

• Open-source software defense
• Critical infrastructure protection
• Smart contract audit acceleration

Aardvark GPT-5 agent expands via private beta. Full EVMbench framework public.

Strategic Implications

Smart contract audits secure billion-dollar treasuries. AI agents could compress multi-week contests to hours. Economic incentives drive rapid capability gains.

Paradigm investment thesis: AI security transforms $100B+ DeFi risk management.

Enterprise Adoption Path

Blockchain teams gain immediate value.

Immediate actions:

• Deploy EVMbench against internal audit pipelines
• Benchmark custom agents on proprietary contracts
• Integrate detect mode into CI/CD vulnerability gates
• Use exploit mode for red team validation

Long-term architecture:

Pre-deploy: AI audit + human review
Post-deploy: Continuous agent monitoring
Incident response: Automated exploit simulation

FAQ