数据集 / Fineweb-Edu-Chinese-V2.1

Fineweb-Edu-Chinese-V2.1 已完整同步

Chinese Fineweb Edu Dataset V2.1 [中文] [English]

[OpenCSG Community] [👾github] [wechat] [Twitter]

📖Technical Report

The Chinese Fineweb Edu Dataset V2.1 is an enhanced version of the V2 dataset, designed specifically for natural language processing (NLP) tasks in the education sector. This version introduces two new data sources, map-cc and opencsg-cc, and retains data with scores ranging from 2 to 3. The dataset entries are organized into different folders based on their scores, allowing for flexible selection of data according to time and computational power requirements during training.

Expanded Data Sources

Key Features

  1. New Data Sources:
    • map-cc
    • opencsg-cc
  2. Score-Based Data Organization:
    • Data entries are categorized into different folders based on their scores:
      • 4-5: High-quality educational content with clear and coherent writing.
      • 3-4: Suitable educational content with some minor issues in coherence or relevance.
      • 2-3: Potentially useful educational content with notable limitations.
  3. Data Volume:
    • 4-5: 70 GB, approximately 46 billion tokens, 17,790,513 lines.
    • 3-4: 800 GB, approximately 530 billion tokens, 289,975,835 lines.
    • 2-3: 1.4 TB, approximately 930 billion tokens, 649,842,063 lines.
  4. Flexible Training:
    • The dataset organization allows for selective use of data based on the available time and computational resources.
    • Researchers and developers can choose specific score ranges to train their models, optimizing for different scenarios.

Data Distribution by Score

score: 4-5

score: 3-4

score: 2-3

We warmly invite developers and researchers interested in this field to follow and engage with the community, working together to advance the technology. Stay tuned for the open-source release of the dataset!

License Agreement

Usage of the Chinese Fineweb Edu dataset requires adherence to the OpenCSG Community License. The Chinese Fineweb Edu dataset supports commercial use. If you plan to use the OpenCSG model or its derivatives for commercial purposes, you must comply with the terms and conditions outlined in the OpenCSG Community License as well as the Apache 2.0 License. For commercial use, please send an email to lorraineg@opencsg.com and obtain permission.

📖Technical Report

Chinese Fineweb Edu V2.1数据集介绍

[OpenCSG 社区] [👾github] [微信] [推特]

**Chinese Fineweb Edu Dataset V2.1** 是 V2 数据集的增强版本，

3 个文件

浏览文件

数据集版权信息

本数据集的许可证为 Apache License 2.0。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证