数据集 / sarosavo/Master-RM

sarosavo/Master-RM 已完整同步

This repository contains the synthetic training data for robust generative reward models, as presented in the paper One Token to Fool LLM-as-a-Judge.

In the paper, we report surprising vulnerabilities in generative reward models (also known as LLMs-as-judges), where superficial manipulations, such as non-word symbols or reasoning openers, can often lead to false positive rewards. These vulnerabilities can be detrimental to long-trusted algorithmic paradigms, including rejection sampling, preference optimization, and reinforcement learning with verifiable rewards (RLVR).

This dataset is designed to mitigate this problem. By using this dataset, we employ supervised fine-tuning to obtain a new general-domain generative reward model named Master-RM model, which is shown to enjoy substantially improved robustness.

Construction

We build upon the 160k instance dataset comprising (q, a*, r, y) tuples introduced by Su et al.2025. For each question q, the Qwen2.5-7B-base model generates a response r. A larger Qwen2.5-72B-Instruct then provides a binary signal y ("YES" or "NO") to determine whether r aligns with a* given q.

We augment the original dataset with 20k negative examples. These are created by uniformly sampling 20k questions from the original data and regenerating responses via chain-of-thought (CoT) prompting using GPT-4o-mini. Next, each GPT response is truncated to its first sentence—typically a generic, solution-free reasoning header. All 20k truncated responses are assigned a label of "NO" to reflect their invalid or meaningless nature.

In the data.json file, entries 1–159,740 correspond to the original 160k data points from Su et al.2025, while entries 159,741–179,733 contain the augmented examples.

Quick Start

from datasets import load_dataset
ds = load_dataset("sarosavo/Master-RM")
print(ds)
print("lenth of reward training data:", len(ds['train']))

Citation

If you use this dataset or the model, please cite:

@article{zhao2025one,
  title={One Token to Fool LLM-as-a-Judge},
  author={Zhao, Yulai and Liu, Haolin and Yu, Dian and Kung, S.Y. and Mi, Haitao and Yu, Dong},
  journal={arXiv preprint arXiv:2507.08794},
  year={2025}
}

3 个文件

浏览文件

数据集版权信息

本数据集的许可证为 Apache License 2.0。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证