A proposal to identify high-impact capabilities of general-purpose AI models : collection of external scientific studies on general-purpose AI models under the EU AI Act

This report proposes a scientific methodology to identify high-impact capabilities in general-purpose AI (GPAI) models, defined in the EU AI Act as capabilities of the most advanced GPAI models. High-impact capabilities play an important role in the EU AI Act since GPAI models with high-impact capabilities are classified as GPAI models with systemic risk. The approach presented by this report is based on observational scaling laws using Principal Components Analysis (PCA) from a set of existing benchmarks, allowing for the extraction of a low-dimensional capability measure that can be used to identify models with high-impact capabilities through setting a suitable threshold. The proposed method involves selecting a diverse set of benchmarks that measure general capabilities, such as MMLU-Pro, GPQA-diamond, MATH-level-5, and HumanEval, and aggregating their scores using a weighted threshold-based metric. The weights are determined by the PCA approach, and the threshold is based on a reference model, to be set by the enforcement authority based on legal, policy, and risk considerations. The report also discusses additional considerations, including proposing for a multi-disciplinary expert group to oversee benchmark selection, and for the approach to be updated every 6 months to account for rapid developments in AI, and suggesting mitigation measures to prevent companies from strategically underperforming on benchmarks. By providing a practical and robust way to assess high-impact capabilities, this methodology aims to contribute to the development of a more comprehensive approach to evaluating GPAI models.