Dataset Card for AgentIF-OneDay

Dataset Details

AgentIF-OneDay is a comprehensive benchmark designed to evaluate AI agents on diverse, daily tasks across work, life, and learning scenarios. Unlike evaluations focused solely on task difficulty, this dataset emphasizes the breadth of general user needs, requiring agents to handle complex attachments, infer implicit instructions, and deliver tangible file-based outputs. It comprises 104 tasks structured around Open Workflow Execution, Latent Instruction, and Iterative Refinement.

Data Structure

Each instance in the dataset is a JSON object with the following fields:

Fields

question_id (string): Unique identifier for the task (e.g., "taskif_1", "taskif_2")
title (string): Brief title summarizing the task
description (string): Detailed task description including step-by-step instructions and requirements
attachment_filenames (list of strings): List of attachment file names associated with the task (may be empty)
score_criteria (list of objects): List of scoring criteria objects, each containing:
- content (string): Description of the criterion
- score (integer): Points awarded (positive) or deducted (negative) for this criterion
reference_answer_description (string): Description or example of the expected answer (may be empty)
reference_answer_attachment_filenames (list of strings): List of reference answer attachment file names (may be empty)
task_tag (string): Category of task execution type. Possible values:
- "Open Workflow Execution"
- "Iterative Refinement"
- "Latent Instruction Inference"
domain_tag (string): Domain category. Possible values:
- "Work"
- "Life"
- "Study"
rubrics_tag (string): Evaluation focus area. Possible values:
- "Content"
- "Form"
- "Execution"
time (string): Expected time to complete the task. Possible values:
- "<1h" (less than 1 hour)
- "2h" (2 hours)
- "4h" (4 hours)
- "8-24h" (8 to 24 hours)

Example

{
  "question_id": "taskif_1",
  "title": "NVIDIA Financial Data Analysis",
  "description": "I am currently conducting a business analysis of NVIDIA and need you to complete the following steps...",
  "attachment_filenames": [],
  "score_criteria": [
    {
      "content": "[nvidia_finance_analysis.xlsx] Correct integration of 2020-2025 data",
      "score": 1
    },
    {
      "content": "[nvidia_finance_analysis.xlsx] Correctly determines that Gross Margin should be calculated",
      "score": 1
    }
  ],
  "reference_answer_description": "",
  "reference_answer_attachment_filenames": [
    "taskif_1 + answer + nvidia_finance_analysis.xlsx"
  ],
  "task_tag": "Open Workflow Execution",
  "domain_tag": "Work",
  "rubrics_tag": "Content",
  "time": "2h"
}

Dataset Statistics

Total instances: 104 tasks
Task categories: 3 types (Open Workflow Execution, Iterative Refinement, Latent Instruction Inference)
Domains: 3 categories (Work, Life, Study)
Time complexity: Ranges from <1 hour to 8-24 hours

Contact

For questions or feedback, please contact: team@xbench.org

Citation

If you use this dataset in your research, please cite our paper:

@misc{chen2026agentifoneday, 
      title={AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios}, 
      author={Kaiyuan Chen and Qimin Wu and Taiyu Hou and Tianhao Tang and Xueyu Hu and Yuchen Hou and Bikun Li and Chengming Qian and Guoyin Wang and Haolin Chen and Haotong Tian and Haoye Zhang and Haoyu Bian and Hongbing Pan and Hongkang Zhang and Hongyi Zhou and Jiaqi Cai and Jiewu Rao and Jiyuan Ren and Keduan Huang and Lucia Zhu Huang and Mingyu Yuan and Naixu Guo and Qicheng Tang and Qinyan Zhang and Shuai Chen and Siheng Chen and Ting Ting Li and Xiaoxing Guo and Yaocheng Zuo and Yaoqi Guo and Yinan Wang and Yinzhou

xbench/AgentIF-OneDay 已完整同步