Rick Muller, director of the Intelligence Advanced Research Projects Activity, said IARPA considers large language models a major focus area for its next artificial intelligence cybersecurity research program, Federal News Network reported Wednesday.
“What we want to be able to do is understand in the next round, what kind of training skews are brought into a large language model that might give unintended consequences? What type of hallucinations are going on?” Muller said Tuesday at an Intelligence and National Security Alliance-hosted event.
“And then how can we make sure that those models can be trained on classified data and not spew out that data if you ask them nicely?” he continued. “If you read the literature in jailbreaking large language models, sometimes it really just takes asking them in the right way.”
IARPA is now considering the next round of AI research as its current program, TrojAI, is set to conclude this year.
What Is TrojAI?
Launched in 2019, TrojAI is an IARPA program that seeks to defend AI systems from malicious attacks, known as Trojans, by conducting research and developing technology to detect such attacks in a completed AI system.
The program aims to deliver software that can quickly and accurately detect Trojans in AI tools before deployment. It has focused research on various AI domains, including image classification, reinforcement learning and natural language processing.
In September 2023, IARPA hosted a challenge with the National Institute of Standards and Technology to prevent malicious actors from manipulating data used to train AI systems.
Register now for the Potomac Officers Club’s 2025 Cyber Summit on May 15. Listen to experts as they discuss new cyber policies, modernization strategies and other trends shaping the cyber domain.
Related Articles
Nextgov/FCW reported Friday that Jane Rathbun, the Department of the Navy’s chief information officer, is retiring after a more than three-decade career in national security. Join the conversations about the latest naval tech capabilities, policies and more at the Potomac Officers Club’s 2025 Navy Summit on Aug. 26. Reserve your seat for this key GovCon networking event! “It is with gratitude, pride, and humility that I close out this chapter of my career as a civil servant,” Rathbun, a two-time Wash100 awardee, said in a LinkedIn post published Thursday. As the Navy CIO, she aligns IT investment priorities with the
David Cattler, director of the Defense Counterintelligence and Security Agency, said DCSA’s partnership with the Department of Defense, industry and academia helps ensure a trusted federal and industrial workforce and protect the country’s trusted workspaces. “We are the gatekeepers, working with you and industry to ensure classified information and technology remains safe from unauthorized foreign access,” Cattler said of DCSA during his keynote address at a conference held Oct. 18. Hear U.S. military leaders and industry experts as they discuss international partnerships, coalition warfare, technological advances and more at the Potomac Officers Club’s GovCon International Summit. Save your spot now
The Cybersecurity and Infrastructure Security Agency has released a draft of the updated Minimum Elements for a Software Bill of Materials, opening a public comment period as it works to strengthen transparency in the software supply chain. Evolving Standards for SBOM The draft builds on the 2021 SBOM Minimum Elements published by the National Telecommunications and Information Administration and reflects advances in tooling, usage and adoption of SBOM practices across government and industry. By incorporating such advancements, the new version raises expectations for how software components are documented and shared. New challenges, vulnerabilities and concerns arise amid the growing global