AI Insight
Researchers developed PHMForge, a benchmark system to evaluate whether AI language models can reliably predict industrial equipment failures using specialized diagnostic tools. Testing across 99 expert-authored scenarios covering engines, rotating equipment, and batteries, the best AI configuration achieved 80.8% accuracy, with most errors stemming from poor task planning rather than tool usage. A critical finding showed that traditional retrieval methods collapsed performance from 100% to 20% on battery diagnostics, demonstrating that AI agents need direct computational access rather than text-based information retrieval for prognostic tasks.
Why it matters
This research addresses whether AI systems can be trusted for safety-critical industrial maintenance decisions, potentially enabling more reliable predictive maintenance in manufacturing, aviation, and energy storage. The findings reveal that current frontier AI models are better at using diagnostic tools than knowing when and how to orchestrate them, identifying specific technical gaps that must be addressed before deployment in high-stakes industrial environments.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: LLM agents are beginning to invoke industrial asset-management tools through the Model Context Protocol (MCP), yet whether they can act reliably on this substrate for safety-critical emph{Prognostics and Health Management (PHM)} is unanswered. Prior benchmarks conflate protocol fluency with reasoning, instrumentation failures with agent failures, and tool use with tool retrieval. We introduce textbf{PHMForge}, an evaluation environment that closes each conflation. PHMForge ships 99 SME-authored scenarios across eight industrial asset classes spanning rotating equipment, aero-engines, and lithium-ion cells, on public datasets including NASA PCoE, served through 39 MCP-native tools wrapping published PHM algorithms (C-MAPSS, ISO~10816, Arrhenius capacity-fade models, time-series foundation models). Krippendorff’s $alpha in [0.74,,0.82]$ on a 30-scenario stratified rotating-equipment/aero-engine sample; the battery extension is single-rater. Across three agentic frameworks and six LLM backbones, the strongest configuration reaches textbf{80.8% pass@1}, with the residual gap concentrated in orchestration and tool-sequencing errors. Crucially, an architectural ablation shows that replacing MCP execution with text-based Retrieval-Augmented Generation (RAG) over telemetry-equivalent evidence collapses Remaining Useful Life emph{pass-all-3} from textbf{100% to 20%} (5/5 vs. 1/5) on the battery class, exposing the structural limits of static retrieval for prognostic computation. Trajectory decomposition shows orchestration errors dominate failures across backbones, while schema-invalid tool calls concentrate in smaller open-weight models. Frontier LLMs are stronger at calling tools than at planning when to call them. PHMForge is open-sourced with deterministic evaluators, a public leaderboard, and a datasheet.