- The National Institute of Standards and Technology has introduced the Artificial Intelligence Technology Evaluation initiative
- Led by the Technology Test and Evaluation Division, AITE provides volunteer-based testing to data providers and AI model developers
- The initiative is intended to serve as a foundation to accelerate AI innovation and real-world application
The National Institute of Standards and Technology announced Monday that its Technology Test and Evaluation Division is rolling out the Artificial Intelligence Technology Evaluation initiative, designed to give researchers a secure testbed for assessing how AI models perform across a range of real-world tasks, datasets, modalities and domains.
The initative is designed to speed up AI innovation and effectiveness, helping AI reach its potential to improve everyday people’s quality of life. Testing under AITE will begin this summer and continue in four phases.
On October 29, be part of the 2026 FedCiv Summit to hear senior government and industry leaders discussing AI adoption, infrastructure resilience, and the priorities shaping the federal civilian market. Register today to secure your seat!
What Are the Phases of AITE?
AITE offers volunteer-based testing to data providers and AI model providers looking to compare their technologies against blind data. The sequestered structure is intended to reduce the risk of train/test data contamination, helping ensure that performance assessments remain rigorous and objective.
AITE’s initial phase centers on three tasks, all focused on image analysis using large vision language models. These tasks span the domains of quantum science, genomics and public safety. In phase one, AITE will accept only a very limited number of additional AI application tests and models, focusing on exercising and refining the processes for collaborating with external partners to develop tests and evaluate external models.
Phase two, the early growth and innovation phase, will see AITE accept a small number of additional tests and models, with an emphasis on expanding the range of tasks, modalities, domains, and corresponding evaluation techniques.
The third phase of AITE involves a large number of additional tests and models, concentrating on scaling both the volume of tests and models and the evaluation techniques used to assess them. The fourth and final phase will focus on sustaining long-term growth and keeping pace with emerging needs.
What Does AITE Participation Entail?
Participation in AITE is structured around two distinct tracks. Data providers may contribute an original dataset from their domain, along with a defined task to be performed using that data, while keeping the dataset inaccessible to other participants. In exchange, data providers receive detailed performance measurements showing how top-performing models handle their specific data and task.
Model providers, meanwhile, submit AI models for evaluation against the available datasets and tasks. This track allows model developers to track how their systems perform across a growing library of datasets and tasks, and to see how their results stack up against other models using identical metrics. NIST said this approach is designed to improve comparability across models while ensuring that evaluation data is never used to train any submitted model.
