Modern chips are far more complex than a decade ago, and design teams face growing pressure to deliver reliable silicon on tight schedules. LLMs offer promising assistance—from drafting RTL to explaining verification failures—but today’s capabilities vary widely across tasks. A common, transparent way to measure progress helps the entire ecosystem learn what works, compare approaches fairly, and focus investment where it matters. Benchmarks also make it easier for new participants to contribute, accelerating practical innovation while keeping expectations grounded. Although LLMs sometimes fall short of expectations—especially on corner cases, reproducibility, and rigorous verification—clear metrics and open evaluation can steadily improve outcomes. The LLM Benchmarking Coalition’s corporate and academic members will build upon the RTL design and verification benchmarks provided in CVDP by extending problems to cover new categories and design domains, monitoring a leaderboard of results, refining benchmarking methods and metrics, and providing an interpretation of the results.
“Experience shows that rigorous benchmarking is the connective tissue between scientific validity and innovative progress. It ensures that claims are credible, comparisons are fair, and advances are real.”
– Igor Markov, Distinguished architect at Synopsys, IEEE Fellow and the coalition vice-chair.
“The Benchmarking Coalition is built on these very principles, ensuring that advances in chip and system design are grounded in evidence, transparency, and measurable impact.”
– Nate Pinckney, Senior Research Scientist at Nvidia and the coalition chair.
Contact us for more information
Nate Pinckney, a senior research scientist at Nvidia, was elected by Coalition members to become the coalition’s first chair. His work covers high-level synthesis methodologies, low-power VLSI design, and cryptographic accelerators, with over 40 publications in these fields. Recently, his research has focused on evaluating large language models and AI agents, including co-creating the VerilogEval and the open-source Comprehensive Verilog Design Problems benchmarks.
Igor Markov is a distinguished architect at Synopsys and an IEEE Fellow, was elected by Coalition members to become the coalition’s vice chair. He previously worked at Meta on AI platforms and at Google on Search. As a professor at the University of Michigan, he performed research on EDA, coauthored a textbook on physical design and co-edited a two-volume EDA Handbook. He won best paper awards at several conferences and published over 200 peer-reviewed papers. At Synopsys, he leads the AI Disruption Task Force.
