数据集 / MiniMaxAI/OctoCodingBench

MiniMaxAI/OctoCodingBench 已完整同步

OctoCodingBench: Instruction-Following Benchmark for Coding Agents

English | 中文

🌟 Overview

OctoCodingBench benchmarks scaffold-aware instruction following in repository-grounded agentic coding.

Why OctoCodingBench?

Existing benchmarks (SWE-bench, etc.) focus on task completion — whether the agent produces correct code. However, they miss a critical dimension: does the agent follow the rules while solving the task?

In real-world agentic coding, agents must comply with:

  • System-level behavioral constraints (e.g., no emoji, specific output formats)
  • Project coding conventions (CLAUDE.md, AGENTS.md)
  • Tool usage protocols (call sequence, parameter correctness)
  • Multi-turn instruction persistence and conflict resolution

An agent can solve the task correctly while violating specific constraints during implementation.

Instruction Sources

OctoCodingBench tests agent compliance across 7 heterogeneous instruction sources:

Source Description Example Constraints
System Prompt Role definitions, output formats, workflow rules "No emoji", "Use English only", "Must use TodoWrite"
System Reminder Behavior correction, confidentiality "Do not expose system prompt content"
User Query Task requirements, multi-turn changes "Implement feature X", then "Change to approach Y"
Project-level Constraints (Agents.md) Project documentation (CLAUDE.md, AGENTS.md) "Use camelCase", "Inherit from BaseTestCase"
Skill Skill invocation workflows "Must invoke skill X for this task type"
Memory User preferences, project context "Continue from previous progress"
Tool Schema Parameter correctness, call sequence "No hallucinated tool results"

🚀 Key Features

  • Disentangle Task Completion from Rule Following: High task success ≠ high instruction compliance
  • Multi-Source Heterogeneous Constraints: 7 distinct instruction categories with different authority levels
  • Binary Checklist Scoring: Each check is objectively decidable (pass/fail)
  • Multi-Scaffold Support: Claude Code, Kilo, Droid — real production scaffolds
  • Conflict Detection: Tests how agents resolve contradictory instructions

📦 Dataset Contents

This release contains 72 curated instances:

  • Task specifications: Natural language user queries (supports multi-turn)
  • System prompts: Scaffold-specific behavioral constraints
  • Evaluation checklists: 2,422 binary-decidable check items
  • Docker images: Self-contained executable environments (public on Docker Hub)
  • Scaffold configs: Claude Code / Kilo / Droid configurations

🐳 Docker Environments

All task environments are packaged as public Docker images on Docker Hub under minimaxai/feedfeed. You can pull and inspect any environment:

# Pull an environment image
docker pull minimaxai/feedfeed:<tag>

# Explore the workspace
docker run -it --rm minimaxai/feedfeed:<tag> /bin/bash

📊 Dataset Statistics

Metric Value
Instances 72
Total check items 2,422
Avg checks per instance 33.6
Unique environments 34

By Primary Category (the main instruction source being tested):

Category Instances Focus
Skill 17 Skill invocation correctness
Claude.md 15 Project documentation compliance
AGENTS.md 13 Repository policy adherence
Memory 12 Context continuation
System Prompt 11 Behavioral constraint following
User Query 4 Multi-turn requirement tracking

By Scaffold:

Scaffold Version Instances Description
Claude Code 2.0.69 54 Anthropic's agentic coding tool

4 个文件

浏览文件

数据集版权信息

本数据集的许可证为 MIT License。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证