The Comprehensive Verilog Design Problems (CVDP) is an open-source benchmark for evaluating Large Language Models (LLMs) on RTL (Register Transfer Level) coding and verification tasks. Initial release in June of 2025 included benchmarks for hardware code generation and comprehension. This release expanded prior work by including problems that had broader coverage and greater depth. CVDP included 783 problems across 13 task categories, covering RTL generation, verification, debugging, specification alignment, and technical Q&A authored by experienced hardware engineers. Problems were in both non-agentic and agentic formats.
The latest version called CVDP 1.1.0, released on June 8, 2026, has made the following improvements targeting AI agents:
- Uses the stable versions of opensource tools – cocotb, Yosys, IVerilog
- Cocotb2.x version will help unlock more supported simulators – open-source and commercial
- An initial setup for running evaluations using Claude Code is now included in the CVDP framework
- Includes an additional 100 data points from 40+ open-source repositories having manually placed bugs and larger sized problems. This dataset is designed to evaluate how well AI agents are able to create RTL from a high-level description.
LLM Benchmarking Coalition: The Silicon Integration Initiative (Si2) is adopting CVDP into their LLM Benchmarking Coalition (LBC), where industry and academic members work together to improve benchmarking and expedite the development of high-quality large language models for semiconductor design problems. The coalition builds upon CVDP by extending problems to cover new categories and design domains, refining benchmarking metrics, and maintaining a leaderboard. Learn more at Si2 LLM Benchmarking Coalition.