数据集 / MiniMaxAI/VIBE

MiniMaxAI/VIBE 已完整同步

VIBE: Visual & Interactive Benchmark for Execution in Application Development

[English] | 中文

🌟 Overview

VIBE (Visual & Interactive Benchmark for Execution) sets a new standard for evaluating Large Language Models (LLMs) in full-stack software engineering. Moving beyond recent benchmarks that rely on static screenshots or rigid workflow snapshots to assess application development, VIBE pioneers the Agent-as-a-Verifier (AaaV) paradigm to assess the true "0-to-1" capability of constructing production-ready applications.

By deploying intelligent agents into dynamic, containerized sandboxes, VIBE performs a hierarchical evaluation across three critical dimensions that directly mirror its name:

  1. Execution (The Foundation): Verifying that the generated project compiles, builds, and launches successfully without fatal errors.
  2. Interactive (The Core): Ensuring all user requirements are met and the business logic remains robust during active agent operation.
  3. Visual (The Apex): Quantify the aesthetic qualities of the user interface, such as visual effects and layout consistency.

🚀 Key Features

  • Agent-as-a-Verifier (AaaV): A novel evaluation framework where vision-capable agents act as autonomous QA testers. They navigate the UI, click buttons, and judge the "look and feel" against human design standards.
  • True Full-Stack Coverage: Beyond standard Web/Backend tasks, VIBE targets often-neglected domains including Native Android & iOS development and high-fidelity Scientific Simulations (Physics/Chemistry/CS).
  • Multi-Dimensional Scoring: We evaluate applications based on a comprehensive reward system:
    • Execution: Does it build and run without crashing?
    • Interaction: Is the logic robust under user inputs?
    • Aesthetics: Is the UI layout professional and visually coherent?

📦 What's Included in This Dataset

This repository contains the foundational data for the VIBE benchmark:

  • 200 Curated Tasks: High-quality prompt specifications ranging from simple tools to complex full-stack applications.
  • Structured Metadata: Detailed difficulty labeling and domain categorization.
  • Evaluation Criteria: (Coming soon) The rubric used by our agent verifiers.

📅 Roadmap

  • Phase 1: Benchmark query prompts & task specifications (Released: December 23, 2025)
  • Phase 2: Containerized sandbox environments & Docker images (Expected: January 2026)
  • Phase 3: Open-source Agent-Verifier scripts & Scoring pipeline (Expected: January 2026)

🧩 Subsets

Subset Description
Web Frontend apps with high aesthetic standards and complex DOM interactions
Simulation Scientific simulations (Physics, Chemistry, CS) requiring high-fidelity rendering
Android Native Android development (Kotlin/Java)
iOS Native iOS development (Swift/Objective-C)
Backend Server-side systems focusing on API integrity and architecture

📊 Dataset Statistics

Subset Easy Medium Hard Total
Web 13 14 13 40
Simulation 13 14 13 40
Android 13 14 13 40
iOS 13 14 13 40
Backend 13 14 13 40
Total 65 70 65 200

📝 Data Format

Each task is a JSON object with the following fields:

{
  "idx": 1,
  "query": "Design and build a portfolio site for a top-tier design agency...",
  "domain": "web",
  "difficulty": "easy"
}
Field Description
idx Unique task identifier
query Natural language requirement specification
domain One of: web, simulation, android, ios, backend
difficulty One of: easy, medium, hard

💻 Dataset Usag

5 个文件

浏览文件

数据集版权信息

本数据集的许可证为 MIT License。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证