Academic center · Science policy · Stanford University, Stanford · Est. 2021
The Stanford Center for Research on Foundation Models (CRFM) is an interdisciplinary initiative at Stanford’s Institute for Human-Centered Artificial Intelligence (HAI) focused on advancing the study, development, and deployment of foundation models from both technical and societal perspectives. CRFM’s work spans core technical foundations (e.g., data, training, systems, inference, interpretability, evaluation), beneficial foundation-model applications in specialized domains (including law, music, robotics, and biomedicine), and societal impact areas such as transparency, supply chains, openness, copyright, privacy, and systemic risk, as well as AI policy engagement to support evidence-based policymaking across jurisdictions.
A flagship CRFM program is the Foundation Model Transparency Index (FMTI), which evaluates foundation-model developers against a structured set of transparency indicators and releases detailed transparency reports.
No curated programs yet. Check the careers page ↗.
We present HELM Arabic Enterprise, a leaderboard for transparent, reproducible evaluation of large language models on Arabic-language benchmarks designed around enterprise use cases. The leaderboard was developed in collaboration with Arabic.AI and builds on the HELM evaluation m
HELM ArabicAs part of our efforts to better understand the multilingual capabilities of large language models (LLMs), we present HELM Arabic, a leaderboard for transparent and reproducible evaluation of LLMs on Arabic language benchmarks. This leaderboard was produced in collaboration with
HELM Long ContextReliable and Efficient Amortized Model-Based EvaluationTLDR: We enhance the reliability and efficiency of language model evaluation by introducing IRT-based adaptive testing, which has been integrated into the HELM framework.
Surprisingly Fast AI-Generated Kernels We Didn’t Mean to Publish (Yet)TL;DR
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity SystemsHELM Capabilities: Evaluating LMs Capability by CapabilityGeneral-Purpose AI Needs Coordinated Flaw ReportingHELM Safety: Towards Standardized Safety Evaluations of Language Models*Work done while at Stanford CRFM
Advancing Customizable Benchmarking in HELM via Unitxt IntegrationThe Holistic Evaluation of Language Models (HELM) framework is an open source framework for reproducible and transparent benchmarking of language models that is widely adopted by academia and industry. To meet HELM users’ needs for more powerful benchmarking features, we are prou
CRFM describes HELM Arabic Enterprise, a leaderboard for transparent, reproducible evaluation of large language models on Arabic-language enterprise benchmarks designed around enterprise use cases (e.g., financial reasoning and legal question answering), developed in collaboration with Arabic.AI and built on HELM evaluation methodology.
HELM ArabicCRFM publishes HELM Arabic, a leaderboard for transparent, reproducible evaluation of LLMs on Arabic language benchmarks produced in collaboration with Arabic.AI, including multiple benchmark components used for evaluation.
HELM Long ContextCRFM releases/announces work on HELM Long Context, positioned around evaluating long-context capabilities within the HELM evaluation framework.
Reliable and Efficient Amortized Model-Based EvaluationCRFM announces a new approach to reliable and efficient amortized model-based evaluation, described as part of its ongoing HELM evaluation work.
Surprisingly Fast AI-Generated Kernels We Didn’t Mean to Publish (Yet)CRFM posts about surprisingly fast AI-generated kernels, presented as related work in its broader research engineering and evaluation ecosystem.
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity SystemsCRFM publishes BountyBench, described as evaluating the dollar impact of AI agent attackers and defenders on real-world cybersecurity systems.