Join us in Santa Clara, CA for the Second Si2 LLM Benchmarking Coalition Workshop: Aligning Agentic Benchmarking and Verification Methods with Real-World EDA Workflows
Semiconductor design and manufacturing companies are building agentic flows/systems on the promise of improved productivity and reuse. But for an industry which differentiates with small changes at the margins of designs, what are the real-world benefits of agentic flows/systems? How can we better understand and take advantage of the paradigm-changing leap from current EDA and manufacturing tool flows to agentic flows/systems? How to build on the decades of understanding and trust in EDA and manufacturing tool flows which have allowed development of CPUs, GPUs, TPUs, and an amazing variety of memory technologies to develop the very AI which is being used to design it?
We’ve invited keynote speakers and discussion facilitators to focus on real-world benchmarking and metrics for verification of agentic flows and systems, exploring examples from outside and inside the semiconductor industry. Many of the LBC discussions use the Comprehensive Verilog Data Problems (CVDP) public test harness and datasets as a baseline (https://github.com/NVlabs/cvdp_benchmark).
The Second Si2 LBC Workshop topics will include:
- Agent system abilities to solve real-world chip design tasks and align with production industry flows
- Importance of verifier and scoring mechanisms over simply expanding datasets
- Adding important score metrics to the LBC-bench public leaderboard
- Accounting for an imperfect test bench when benchmarking agentic EDA flows
- Challenges with self-correcting agent interaction with a test bench
Workshop is open to all interested members from Si2 LBC industry companies and universities, and anyone wanting to learn more about the LBC activities.
Detailed agenda coming soon!
Second Si2 LLM Benchmarking Coalition Workshop
- Wednesday, July 29, 8:00 AM – 6:30 PM Pacific Time
- Location: Keysight Technologies, 5301 Stevens Creek Blvd Santa Clara, CA 95051
- Registration required – complimentary registration, lunch and coffee breaks
- Registration link-> https://survey.zohopublic.com/zs/awBBd2
About the Si2 LLM Benchmarking Coalition:
The Si2 LLM Benchmarking Coalition is an effort under Silicon Integration Initiative (Si2) focused on advancing empirical evaluation for AI in chip design.
The goal of the coalition is to bring together high-quality AI-for-chip design datasets into a consistent, reproducible format, operate a centralized leaderboard maintained by Si2 staff, and, as a community, establish reporting standards for how these benchmarks are used in the literature. More broadly, we aim to align the field around shared evaluation, improving benchmarks collaboratively rather than having them compete in isolation. You can find more details here: https://si2.org/llm-benchmarking-coalition/.
About Si2
The Silicon Integration Initiative Inc. is a global, not-for-profit membership organization with approximately 70 corporate members who collaborate on trusted standards and shared solutions that reduce development costs and increase design productivity for semiconductor foundries, fabless design companies, and EDA software providers. Learn more at https://si2.org/.
Questions? Contact Leigh Anne Clevenger, Vice President, Technology, Si2, le*****************@*i2.org