ChemRAGBench: Evaluating Retrieval-Augmented
Chemical Reasoning

Abstract: Large language models (LLMs) have shown transformative capabilities in computational reasoning across scientific domains. In chemistry, they offer potential for property prediction, mechanistic reasoning, and multi-source data integration. However, existing benchmarks emphasize factual recall over retrieval-augmented reasoning. ChemRAGBench is introduced as a scalable benchmark spanning six chemical domains—organic, inorganic, medicinal, computational, physical, and environmental—with difficulty-scaled questions ranging from factual retrieval to cross-domain synthesis. Evaluating four contemporary LLMs (ChatGPT, Gemini, DeepSeek, Grok) reveals strong fluency and clarity but persistent gaps in mechanistic reasoning and evidence integration. By framing retrieval-augmented chemical reasoning as an optimization problem, ChemRAGBench highlights avenues for improving learning strategies, retrieval pipelines, and multi-step reasoning, providing a reproducible framework for advancing AI-driven chemical reasoning.