☰ Menu
× Close

LLM Benchmarking Coalition

Expediting the development of high-quality LLMs for semiconductor design

The LLM Benchmarking Coalition is a newly forming coalition that will expedite the development of high-quality LLMs for semiconductor design by creating relevant and comprehensive evaluation datasets and metrics for semiconductor design problems. The goals of this Coalition are to:

  • Define new datasets/problems to be co-developed with Si2 or contributed by Coalition Members.
  • Decide what data moves into a Public Repo or remains in a Private Repo, and under what terms.
  • Provide Member access to the Private Repo to tune methods and improve benchmarking & LLM development.
  • Extend datasets/problems to cover new categories and design domains.
  • Monitor the public and Si2 Scoreboard results and refine benchmarking methods and metrics.
  • Provide interpretation of results.

Contact us for more information about this rapidly evolving Coalition