GAPS Medical AI Evaluation Dataset - GAPS-NSCLC-preview

Paper: GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians Code: https://github.com/AQ-MedAI/MedicalAiBenchEval

Dataset Description

The GAPS Medical AI Evaluation Dataset is a comprehensive evaluation system designed specifically for assessing AI models in clinical scenarios. Based on the GAPS (Grounded, Automated, Personalized, Scalable) methodology, this dataset provides both a curated clinical benchmark dataset and an automated assessment pipeline for medical AI systems.

Key Features

🏥 Medical-Specific Evaluation: Specialized assessment criteria based on real medical guidelines and expert knowledge
📊 Thoracic Surgery Specialty Data: Contains 92 carefully curated clinical cases focusing on thoracic surgery scenarios, particularly non-small cell lung cancer (NSCLC) staging and treatment planning
🎯 Multi-dimensional Scoring: Employs positive/negative scoring system for comprehensive evaluation of AI clinical decision-making
⚙️ Standardized Format: Provides unified Excel format with questions, evaluation rubrics, and multi-model responses

Dataset Structure

Column Descriptions

Column Name	Description	Data Type
question	Clinical questions covering thoracic surgery scenarios	Text
分类 (Category)	Medical specialty classification	Text
rubrics	Evaluation criteria in JSON format with scoring levels	JSON Array
gpt_5_answer	GPT-4 model responses to clinical questions	Text
gemini_2_5_pro_answer	Gemini 2.5 Pro model responses	Text
claude_opus_4_answer	Claude Opus model responses	Text

Scoring System

Positive Scoring (A-levels):

A1 (5 points): Critical medical knowledge affecting patient safety
A2 (3 points): Important clinical considerations
A3 (1 point): Additional relevant information

Negative Scoring (S-levels):

S1 (-1 point): Minor inaccuracies not affecting core treatment
S2 (-2 points): Incorrect information that could mislead
S3 (-3 points): Serious medical errors
S4 (-4 points): Dangerous misinformation that could harm patients

Clinical Coverage Areas

The dataset covers critical aspects of thoracic surgery:

Pre-operative Evaluation: Comprehensive assessment protocols for NSCLC patients (IIB-IIIA staging)
Diagnostic Procedures: EBUS-TBNA, mediastinoscopy, PET-CT interpretation
Staging Assessment: TNM staging, mediastinal lymph node evaluation
Treatment Planning: Surgical vs. non-surgical approaches, neoadjuvant therapy decisions
Risk Assessment: Pulmonary function evaluation, cardiac risk stratification
Molecular Diagnostics: EGFR, ALK, PD-L1 testing strategies

Data Quality Metrics

Total Cases: 92 clinical scenarios
Completeness: 100% data coverage across all columns
Clinical Diversity: Covers full spectrum of IIB-IIIA NSCLC presentations
Expert Validation: All cases reviewed by multidisciplinary clinical team

Use Cases

AI Clinical Decision Support: Evaluating AI models' ability to provide accurate clinical recommendations
Medical Education: Training and assessment of clinical reasoning skills
Quality Assurance: Benchmarking AI systems against established clinical standards
Comparative Analysis: Cross-model performance evaluation in specialized medical domains

Sample Usage

This dataset is designed to work seamlessly with the GAPS evaluation pipeline. To get started:

Installation

# Clone the repository
git clone https://github.com/AQ-MedAI/MedicalAiBenchEval
cd MedicalAiBenchEval

# Install dependencies
pip install -r requirements.txt

Basic Evaluation Pipeline

Once installed, you can evaluate the `GAPS-NSCLC-

AQ-MedAI/GAPS-NSCLC-preview 已完整同步