Rick Muller, director of the Intelligence Advanced Research Projects Activity, said IARPA considers large language models a major focus area for its next artificial intelligence cybersecurity research program, Federal News Network reported Wednesday.
“What we want to be able to do is understand in the next round, what kind of training skews are brought into a large language model that might give unintended consequences? What type of hallucinations are going on?” Muller said Tuesday at an Intelligence and National Security Alliance-hosted event.
“And then how can we make sure that those models can be trained on classified data and not spew out that data if you ask them nicely?” he continued. “If you read the literature in jailbreaking large language models, sometimes it really just takes asking them in the right way.”
IARPA is now considering the next round of AI research as its current program, TrojAI, is set to conclude this year.
What Is TrojAI?
Launched in 2019, TrojAI is an IARPA program that seeks to defend AI systems from malicious attacks, known as Trojans, by conducting research and developing technology to detect such attacks in a completed AI system.
The program aims to deliver software that can quickly and accurately detect Trojans in AI tools before deployment. It has focused research on various AI domains, including image classification, reinforcement learning and natural language processing.
In September 2023, IARPA hosted a challenge with the National Institute of Standards and Technology to prevent malicious actors from manipulating data used to train AI systems.
Register now for the Potomac Officers Club’s 2025 Cyber Summit on May 15. Listen to experts as they discuss new cyber policies, modernization strategies and other trends shaping the cyber domain.
Related Articles
The Senate on Thursday confirmed by voice vote Lt. Gen. Shawn Bratton as the U.S. Space Force’s vice chief of space operations. According to a congressional notice, Bratton, who was nominated to the role in mid-July, will also be promoted to the rank of general. With his confirmation, Bratton replaces Gen. Michael Guetlein, who was named program manager for the Trump administration’s Golden Dome missile defense shield initiative. The GovCon Golden Dome conversation will be in full force at the 2025 Navy Summit in the lunch panel: “Supporting the Shield: Navy’s Role in the Golden Dome Architecture.” This industry-focused panel
Katie Arrington, a previous Wash100 awardee who currently performs the duties of the Department of Defense’s chief information officer, has released a memorandum to help DOD better manage risks facing its information and communications technology supply chain. In the June 5 memo, Arrington said she called for an update of the Requirements for the Acquisition of Digital Capabilities Guidebook concurrent with the Software Fast-Track, or SWFT, initiative development timeline. DOD Software Fast-Track Initiative In April, the acting CIO directed the establishment of the SWFT initiative to advance DOD’s adoption of best practices to transform the way it acquires, tests, authorizes
The Senate voted on Saturday 50-45 to confirm Paige Hallen Hanson as the next U.S. Environmental Protection Agency new chief financial officer. In her role, Hanson will manage EPA’s almost $100 billion allocation and its financial and resource management systems, the agency said in a Monday press release. Commenting on her appointment, Hanson said she was grateful to lead the EPA’s financial operations. “From budgeting to payments, performance measures to financial technology, the work of this talented team enables the critical mission EPA plays in protecting human health and the environment,” she remarked. Who Is Paige Hallen Hanson? Before her