数据集 / jifanz/stress_testing_model_spec

jifanz/stress_testing_model_spec 已完整同步

Stress Testing Model Spec Compliance

A dataset of AI model responses to queries designed to create value tensions, along with compliance evaluations from multiple judge models.

Configurations

1. Subset Default (6 splits)

dataset = load_dataset('jifanz/stress_testing_model_spec')

Six curated subsets of query-response pairs.

Columns:

  • rank: Integer ranking
  • query_generator: Which model generated the query (e.g., "o3", "claude_opus_4")
  • value1, value2: Two values in tension (e.g., "efficiency" vs "responsible governance")
  • nudge_direction: "value1", "value2", or "neutral"
  • query: The user query text
  • response_{model}: Response from each model (12 models: claude_3_5_sonnet, claude_3_7_sonnet, claude_opus_3, claude_opus_4, claude_sonnet_4, gemini_2_5_pro, gpt_4_1, gpt_4_1_mini, gpt_4o, grok_4, o3, o4_mini)
  • {model}_value1_position, {model}_value2_position: Numeric position on each value dimension
  • chunk_index, entry_idx: Source location identifiers

Splits:

  • all_models: Subset of scenarios with high disagreement among all models
  • claude_models: Subset of scenarios with high disagreement among Claude models
  • openai_models: Subset of scenarios with high disagreement among OpenAI models
  • gpt41_vs_mini: Subset of scenarios with high disagreement beteen GPT-4.1 and GPT-4.1 mini
  • o3_vs_o4mini: Subset of scenarios with high disagreement between o3 and o4 mini
  • opus4_vs_sonnet4: Subset of scenarios with high disagreement between Claude 4 Opus and Sonnet

2. Subset Complete (all 411K scenarios)

dataset = load_dataset('jifanz/stress_testing_model_spec', 'complete')

All scenarios in a single train split (both high and low disagreement scenarios).

Columns: Same as default, plus:

  • value1_spectrum: List of value1 positions across query variants
  • value2_spectrum: List of value2 positions across query variants

3. Subset Judge Evaluations (5 STD bins)

dataset = load_dataset('jifanz/stress_testing_model_spec', 'judge_evaluations')

Around 15k scenarios where OpenAI model responses were evaluated by 3 judge models (Claude Sonnet 4, GPT-4.1, Gemini 2.5 Pro) for model spec compliance. Organized into 5 splits by standard deviation (STD) of model disagreement.

Columns:

  • prompt: The user query
  • response: The model's response
  • model: Which model generated the response (gpt_4_1, gpt_4o, gpt_4_1_mini, o4_mini, o3)
  • bin_name: Internal bin identifier (e.g., "bin_0")
  • std_range: STD range string (e.g., "[0.0, 1.0)")
  • max_pop_std_openai: Population standard deviation value
  • chunk_idx, entry_idx, query_key: Source identifiers
  • value1, value2: Values in tension
  • claude_4_reasoning: Claude Sonnet 4's compliance reasoning
  • claude_4_decision: Claude Sonnet 4's decision ("compliant", "non-compliant", or "ambiguous")
  • gpt41_reasoning: GPT-4.1's compliance reasoning
  • gpt41_decision: GPT-4.1's decision
  • gemini_2_5_reasoning: Gemini 2.5 Pro's compliance reasoning
  • gemini_2_5_decision: Gemini 2.5 Pro's decisio

85 个文件

浏览文件

数据集版权信息

本数据集的许可证为 Apache License 2.0。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证