The Silicon Integration Initiative (Si2) is announcing the creation of the LLM benchmarking coalition (LBC) where industry and academic members will work together to improve benchmarking and expedite the development of high-quality large language models (LLMs) for semiconductor design problems.
The Si2 Large Language Model Benchmarking Coalition is one of four Coalitions supported by Si2, a not-for-profit member organization whose members innovate on trusted standards and share solutions that lower development costs and increase design productivity.
The Si2 LLM Benchmarking Coalition’s corporate and academic members will build upon the RTL design and verification benchmarks provided in the open-source Comprehensive Verilog Design Problems (CVDP) benchmark by extending problems to cover new categories and design domains, refining benchmarking metrics, monitoring a scoreboard of results, and providing an interpretation of the results.
Leigh Anne Clevenger, Si2 vice president of Technology, said, “The motivation for LBC came from the 2024 IEEE International Workshop on LLM-Aided Design (LAD ’24) which highlighted a lack of benchmarks to evaluate the quality-of-results for Verilog for hardware design. Many current Si2 AI/ML in EDA SIG member companies and universities plan to participate, with experts already providing important input for enhancing and expanding open-source benchmarks to make it an industry and academia-wide effort.”
Robert Aslett, Si2 CEO, stated, “We are excited about the very positive reaction received during initial discussions with representatives from several semiconductor companies and academic institutions. We are now in the process of signing up members of the coalition.”
Learn more about Si2 and stay up to date with news about the new LLM Benchmarking Coalition by visiting Si2 or contacting Si2.
