TLDR:
- ETH Zurich, INSAIT, and LatticeFlow AI released the first EU AI Act compliance evaluation framework for generative AI
- The framework includes a technical interpretation of the EU AI Act and an open-source tool to evaluate Large Language Models
- Initial evaluations of public foundation models from major tech companies were conducted
- The European Commission welcomed this initiative as a step towards implementing the AI Act
- The framework is open for contributions from AI researchers, developers, and regulators
The artificial intelligence landscape is evolving rapidly, and with it comes the need for regulatory frameworks to ensure responsible development and deployment of AI technologies. In a move towards this goal, ETH Zurich, INSAIT, and LatticeFlow AI have unveiled the first evaluation framework for generative AI models in compliance with the European Union’s AI Act.
The framework, accessible at compl-ai.org, provides a technical interpretation of the EU AI Act, translating regulatory requirements into measurable technical benchmarks. This initiative aims to bridge the gap between high-level regulatory guidelines and practical implementation for AI developers and companies.
At the core of this release is an open-source framework designed to evaluate Large Language Models (LLMs) against these technical requirements. The tool has already been used to assess public foundation models from industry leaders such as OpenAI, Meta, Google, Anthropic, and Alibaba, offering insights into their compliance with the EU AI Act.
Thomas Regnier, spokesperson for the European Commission’s digital economy, research, and innovation sector, expressed support for the initiative, stating,
“The European Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements, helping AI model providers implement the AI Act.”
The EU AI Act, while comprehensive in its regulatory approach, does not provide specific technical guidelines for implementation. To address this gap, the European Commission has initiated a consultation on the Code of Practice for providers of general-purpose Artificial Intelligence (GPAI) models.
The newly released COMPL-AI framework is expected to contribute significantly to this process, offering a starting point for the GPAI working groups tasked with supervising the implementation and enforcement of AI Act rules.
The evaluation framework incorporates 27 state-of-the-art benchmarks to assess LLMs against the technical requirements derived from the EU AI Act. This comprehensive approach allows for a detailed analysis of AI models across various compliance criteria.
Professor Martin Vechev, Full Professor at ETH Zurich and Founder and Scientific Director of INSAIT, emphasized the collaborative nature of the project, stating,
“We invite AI researchers, developers, and regulators to join us in advancing this evolving project. We encourage other research groups and practitioners to contribute by refining the AI Act mapping, adding new benchmarks, and expanding this open-source framework.”
The framework’s versatility extends beyond the EU AI Act, as it can be adapted to evaluate AI models against future regulatory acts in different jurisdictions. This flexibility makes it a valuable tool for organizations operating across multiple regulatory environments.
Dr. Petar Tsankov, CEO and Co-Founder of LatticeFlow AI, highlighted the practical implications of the framework, saying,
“With this framework, any company can now evaluate their AI systems against the EU AI Act technical interpretation. Our vision is to enable organizations to ensure that their AI systems are not only high-performing but also fully aligned with the regulatory requirements.”
Initial evaluations using the COMPL-AI framework have revealed varying levels of compliance among major tech companies’ AI models. While some models performed well in certain areas, others showed potential shortcomings in crucial aspects such as cybersecurity resilience and discriminatory output.
