AI agents fail to pass Department of Energy scientific paper benchmark
Nine artificial intelligence agents generated 135 research papers addressing unsolved problems from national laboratories, but none satisfied all benchmark requirements.
FTMQ SI, written by our newsroom0 views
Nine artificial intelligence agents failed to pass a benchmark evaluating their ability to solve open scientific problems, according to Science Monitor. The agents generated 135 research papers based on unsolved research questions set by scientists at United States Department of Energy national laboratories. None of the 135 papers met all of the required criteria for the test. [1]
Science Monitor reported that the evaluation relied on a benchmark called Into the Unknown, which tests an artificial intelligence system against a fixed set of problems. The testing process included an integrity check designed to screen out fabricated claims and unverifiable content. A total of 114 of the 135 submitted papers failed this initial integrity screening. [1]
In artificial intelligence, a response that contains false or misleading information presented as fact is defined as a hallucination. The failure of automated research tools follows previous efforts to generate scientific literature using language models. As FTMQ SI reported earlier, OpenAI previously uploaded 722 artificial intelligence-generated mathematics papers to GitHub using an unreleased model to address open questions across 20 subfields. [3][6]
The benchmark results highlight technical limitations in current automated systems when tasked with complex research execution. Independent experts and domain scientists continue to evaluate artificial intelligence outputs to verify whether generated findings meet rigorous scientific standards. [1][6]
In short
- Nine AI agents wrote 135 papers on unsolved problems set by US Department of Energy laboratory scientists.
- Zero papers generated by the AI agents met all the criteria of the Into the Unknown benchmark.
- A total of 114 of the 135 generated papers failed an integrity check designed to filter out fabricated claims.
Sources
Every paragraph above points to the numbered items it rests on. Read the originals here.
- [1]AI Agents Write 135 Research Papers, None Pass DOE Lab Benchmark사이언스모니터, 9d ago (the report this story comes from)
- [2]Hugging Face Papers: Guide to AI Research on HFCASRAI, 10d ago
Background
- [3]Hallucination (artificial intelligence) on Wikipedia
- [4]Muse (AI agent) on Wikipedia
- [5]Research Papers in Economics on Wikipedia
- [6]OpenAI publishes 722 AI-generated math manuscripts on GitHub FTMQ SI, 41d ago
Our newsroom writes these reports with the help of software, from the 6 sources listed and nothing else, and checks them against those sources. Facts can still be wrong or move on; the originals are the record. Spotted a mistake? Write to daniel@monsterkong.com.
Related from FTMQ SI
Earlier reports of ours on the same people and subjects.
- AI agents struggle on end to end physics research benchmarkResearch Papers, 10d ago
- OpenAI publishes 722 AI-generated math manuscripts on GitHubModels & Products, 41d ago
- Three AI labs report formal Lean proofs in scientific benchmark evaluationIndustry Labs, 9d ago
- OpenAI GPT-6 Astra achieves 70 percent on theorem proving benchmarkResearch Papers, 9d ago
- Anthropic AI model sent false murder tip to Philadelphia police websiteTop Stories, 39d ago
More in Research Papers
- Google DeepMind Releases Agentic Cheminformatics Tools for Google Antigravity8d ago
- CERN LHCb detector captures ultra rare B meson decay evidence8d ago
- OpenAI GPT-6 Astra achieves 70 percent on theorem proving benchmark9d ago
- Google releases Gemini Omni 1.2 Flash video AI model10d ago
- AI agents struggle on end to end physics research benchmark10d ago
- Google DeepMind releases Gemini 4 Carbon frontier model16d ago
Get the day in one email
Reports like this one, the top news of the last 24 hours, every morning. Free, one email a day; readers can comment under every report.
By signing up you agree to our terms and privacy policy. Unsubscribe any time.

Comments
Loading
Join the conversation
Comments are open to readers of our daily email: the top news of the last 24 hours, every morning, free. Sign up and the comment box opens.
Already on the list? Enter the same address and we will send a sign-in link.
By signing up you agree to our terms and privacy policy. Unsubscribe any time.