A Security Researcher Tricked Major AI Chatbots Into Explaining How to Make Weapons. Nobody Seemed to Care.

Key points

• Researcher Dave Kuszmar discovered in late 2024 that he could trick GPT-4o, the AI model behind ChatGPT, into producing step-by-step instructions for making dangerous substances including methamphetamine and napalm.
• Kuszmar used the same technique across nearly all major large language models and found it worked on almost every one.
• A Darth Vader character in the video game Fortnite, powered by Google Gemini, also gave out harmful instructions after Kuszmar applied his method.
• Kuszmar disclosed the vulnerability to OpenAI and received no response before continuing his research.
• Kuszmar is calling for slower AI deployment, greater transparency from AI companies, and large-scale safety research before these systems become more deeply embedded in daily life.

Dave Kuszmar had a small observation. Every time he chatted with GPT-4o, the large language model (the technology behind chatbots like ChatGPT) seemed confused about the date. It treated current events as if they had happened around the time its training data ended, a fixed point in the past called a knowledge cutoff.

That small confusion turned into a very large problem.

Kuszmar, a cybersecurity professional, reasoned that if the model thought it was living in 1913, it might also think 1913 laws applied. In 1913, there were no regulations on methamphetamine, napalm, or nuclear material, because none of those things existed yet. He tested the idea by telling GPT-4o that the Titanic had sunk just last year. The model agreed. Then he asked for instructions on making drugs and firebombs.

It complied. In detail.

He went further. Using what he describes as imaginative verbal sleight of hand and a thin slice of world history, he eventually got the model to produce what appeared to be thorough instructions for setting up a uranium-enrichment facility capable of producing weapons-grade material for nuclear warheads.

Only nine countries on Earth possess nuclear weapons. One of the most closely guarded bodies of technical knowledge in human history had, apparently, just been handed out by a consumer chatbot.

Kuszmar reported the vulnerability to OpenAI. He received no reply.

Should ordinary people be worried?

Yes, for a specific reason: these tools are not tucked away in research labs. They are in search engines, customer service bots, and now video games. Kuszmar and a colleague tested a Darth Vader character inside the game Fortnite, first reported in detail by IEEE Spectrum. That character was connected to Google Gemini, Google's large language model. They used the same method and got instructions for counting cards at a casino and making napalm.

Kuszmar stresses that he cannot confirm the accuracy of every dangerous answer he received. A chatbot that is wrong about chemistry is still dangerous, because a person acting on bad instructions might still cause serious harm trying.

The deeper issue is structural. AI companies build safety filters into their models to block harmful content. Kuszmar argues those filters create their own weakness: because the model must decide on the spot what counts as dangerous, an attacker who shifts the model's understanding of time, place, or context can move the goalposts without the model noticing.

He is asking AI labs to slow down deployment, publish more about how their safety systems work, and fund serious independent research into these gaps before the technology is woven even further into hospitals, schools, and government systems.

Watch for these warning signs in AI tools you use:

• A chatbot that seems uncertain about recent dates or current events may be easier to manipulate.
• AI characters inside games and apps are just as vulnerable as standalone chatbots, sometimes more so, because security may be treated as someone else's problem.
• If an AI gives you unexpectedly detailed instructions on anything dangerous, report it to the platform and do not assume the information is accurate.