The LLM Benchmarking Coalition is a newly forming coalition that will expedite the development of high-quality LLMs for semiconductor design by creating relevant and comprehensive evaluation datasets and metrics for semiconductor design problems. The goals of this Coalition are to:
- Define new datasets/problems to be co-developed with Si2 or contributed by Coalition Members.
- Decide what data moves into a Public Repo or remains in a Private Repo, and under what terms.
- Provide Member access to the Private Repo to tune methods and improve benchmarking & LLM development.
- Extend datasets/problems to cover new categories and design domains.
- Monitor the public and Si2 Scoreboard results and refine benchmarking methods and metrics.
- Provide interpretation of results.
Contact us for more information about this rapidly evolving Coalition