GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts?

Fan Yuan^1,*, Yuchen Yan^1,*, Yifan Jiang¹, Haoran Zhao¹, Tao Feng¹, Jinyan Chen¹, Yanwei Lou¹,
Wenqi Zhang¹, Yongliang Shen^1,†, Weiming Lu¹, Jun Xiao¹, Yueting Zhuang¹

¹Zhejiang University
^*Equal contribution, ^†Corresponding author

💻 Github | 🤗 Dataset | 🤗 Hf-Paper | 📝 Arxiv | 🌐 ProjectPage

🔔 News

🔥 2025.09.30: Paper is released! 🚀
🔥 2025.09.28: Code for evaluation is available! 🚀
🔥 2025.09.28: Home page is available. 🌟

👁️ Overview

GSM8K-V is a purely visual multi-image mathematical reasoning benchmark that systematically maps each GSM8K math word problem into its visual counterpart to enable a clean, within-item comparison across modalities. Built via an automated pipeline that extracts and allocates problem information across scenes, generates scene-level descriptions, and renders images, coupled with meticulous human annotation, the benchmark comprises 1,319 high-quality multiscene problems (5,343 images) and addresses limitations of prior visual math evaluations that predominantly focus on geometry, seldom cover visualized word problems, and rarely test reasoning across multiple images with semantic dependencies. Evaluations of a broad range of open- and closed-source models reveal a substantial modality gap—for example, Gemini-2.5-Pro attains 95.22% accuracy on text-based GSM8K but only 46.93% on GSM8K-V—highlighting persistent challenges in understanding and reasoning over images in realistic scenarios and providing a foundation to guide the development of more robust and generalizable vision-language models.

Our main contributions are summarized as follows.

We propose an automated framework that converts text-based math word problems into visual form. Specifically, we construct detailed multi-scene textual descriptions and leverage image generation models to produce corresponding visual representations.
Building on the proposed data construction framework and careful human annotation, we introduce a vision-based mathematical reasoning benchmark GSM8K-V, which enables the evaluation of VLMs on more realistic mathematical problem-solving scenarios.
We perform a thorough evaluation and analysis of the existing VLMs in GSM8K-V. The results reveal substantial room for improvement, and our analysis provides valuable insights for enhancing the mathematical reasoning capabilities of future VLMs.

🚀 Sample Usage

# Clone the repository
git clone https://github.com/ZJU-REAL/GSM8K-V.git
cd GSM8K-V

# Create conda environment (optional)
conda create -n gsm8k-v python=3.10
conda activate gsm8k-v

# Install dependencies
pip install -r requirements.txt

# Command for vllm mode
python eval.py --type vllm \
--model_name <eval_model_name> --api_base <vl

5352 个文件

浏览文件

数据集版权信息

本数据集的许可证为 MIT License。如有违反相关条款，请联系 WEHUB，我们将及时处理。查看许可证