☰ Menu
× Close

LLM Benchmarking Coalition

Expediting the development of high-quality LLMs for semiconductor design

Modern chips are far more complex than a decade ago, and design teams face growing pressure to deliver reliable silicon on tight schedules. LLMs offer promising assistance—from drafting RTL to explaining verification failures—but today’s capabilities vary widely across tasks. A common, transparent way to measure progress helps the entire ecosystem learn what works, compare approaches fairly, and focus investment where it matters. Benchmarks also make it easier for new participants to contribute, accelerating practical innovation while keeping expectations grounded. Although LLMs sometimes fall short of expectations—especially on corner cases, reproducibility, and rigorous verification—clear metrics and open evaluation can steadily improve outcomes. The LLM Benchmarking Coalition’s corporate and academic members will build upon the RTL design and verification benchmarks provided in CVDP by extending problems to cover new categories and design domains, monitoring a leaderboard of results, refining benchmarking methods and metrics, and providing an interpretation of the results.

“Experience shows that rigorous benchmarking is the connective tissue between scientific validity and innovative progress. It ensures that claims are credible, comparisons are fair, and advances are real.”

– Igor Markov, Distinguished architect at Synopsys, IEEE Fellow and the coalition vice-chair.

“The Benchmarking Coalition is built on these very principles, ensuring that advances in chip and system design are grounded in evidence, transparency, and measurable impact.”

– Nate Pinckney, Senior Research Scientist at Nvidia and the coalition chair.

Contact us for more information

LBC Governance

Chair: Nathaniel Pinckney – Sr. Research Scientist NVIDIA

Nate Pinckney, a senior research scientist at Nvidia, was elected by Coalition members to become the coalition’s first chair. His work covers high-level synthesis methodologies, low-power VLSI design, and cryptographic accelerators, with over 40 publications in these fields. Recently, his research has focused on evaluating large language models and AI agents, including co-creating the VerilogEval and the open-source Comprehensive Verilog Design Problems benchmarks.

Vice Chair: Igor Markov – Distinguished Architect Synopsys

Igor Markov is a distinguished architect at Synopsys and an IEEE Fellow, was elected by Coalition members to become the coalition’s vice chair. He previously worked at Meta on AI platforms and at Google on Search. As a professor at the University of Michigan, he performed research on EDA, coauthored a textbook on physical design and co-edited a two-volume EDA Handbook. He won best paper awards at several conferences and published over 200 peer-reviewed papers. At Synopsys, he leads the AI Disruption Task Force.

Members

ARM
Arizona State University
Barcelona Supercomputer Center
Cadence Design Systems
Columbia University
Drexel University
Google
IBM
Intel
Keysight
Meta
Normal Computing
NVIDIA
NXP
NC State University
New York University
Samsung
Siemens
Silimate
SK Hynix
Synopsys
UC San Diego