数据集 / cloudcatcher2/VCBench

cloudcatcher2/VCBench 已完整同步

Overview

VCBench provides a standardized framework for evaluating vision-language models. This document outlines the procedures for both standard evaluation and GPT-assisted evaluation of your model's outputs.

1. Standard Evaluation

1.1 Output Format Requirements

Models must produce outputs in JSONL format with the following structure:

{"id": <int>, "pred_answer": "<answer_letter>"}
{"id": <int>, "pred_answer": "<answer_letter>"}
...

Example File (submit.jsonl):

{"id": 1, "pred_answer": "A"}
{"id": 2, "pred_answer": "B"}
{"id": 3, "pred_answer": "C"}

1.2 Evaluation Procedure

  1. Ensure your predictions file follows the specified format
  2. Run the evaluation script: 
    python evaluate_vcbench.py -p ./path/to/predictions.jsonl -g ./path/to/VCBench_with_answer.json

VCBench_with_answer.json is the ground truth file which can be downloaded from here.

2. GPT-Assisted Evaluation

2.1 Output Format Requirements

For natural language responses, use this JSONL format:

{"id": <int>, "pred_answer": "<natural_language_response>"}
{"id": <int>, "pred_answer": "<natural_language_response>"}
...

Example File (nl_predictions.jsonl):

{"id": 1, "pred_answer": "The correct answer is A"}
{"id": 2, "pred_answer": "After careful analysis, option B appears correct"}
{"id": 3, "pred_answer": "C is the right choice"}

2.2 Environment Setup

Set your Dashscope API key:

export DASHSCOPE_KEY="your_api_key_here"

2.3 Evaluation Procedure

python evaluate_vcbench_by_gpt.py -p ./path/to/nl_predictions.jsonl -g ./path/to/VCBench_with_answer.json

3. Expected Output

Both evaluation scripts will provide:

  • Overall accuracy percentage
  • Per-question-type accuracy breakdown
  • Progress updates during evaluation

8423 个文件

浏览文件

数据集版权信息

本数据集的许可证为 Apache License 2.0。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证