Matrix

An open-source pretraining dataset containing 4690 billion tokens, this bilingual dataset with both English and Chinese texts is used for training neo models.

Dataset Composition

The dataset consists of several components, each originating from different sources and serving various purposes in language modeling and processing. Below is a brief overview of each component:

Common Crawl
Extracts from the Common Crawl project, featuring a rich diversity of internet text including websites, blogs, news articles, and more.

Code
A collection of coding-related data.

Paper
Consists of academic and research papers covering a broad spectrum of disciplines, offering technical and domain-specific language.

Book
Comprises texts from a range of published books, encompassing literature, non-fiction, textbooks, and more.

Instruction
Features a collection of texts primarily in a Q&A format.

Exam
Contains various educational materials and assessments used in academic examinations.

News
A collection of texts from various journalistic sources, reporting on current events and news stories.

Wiki
Articles from various encyclopedic sources, not limited to Wikipedia, covering a wide array of topics and information.

Patent
Includes texts from patent documents, providing detailed descriptions of inventions and their applications.

Citation

@article{zhang2024mapneo,
    title   = {MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series},
    author  = {Ge Zhang and Scott Qu and Jiaheng Liu and Chenchen Zhang and Chenghua Lin and Chou Leuang Yu and Danny Pan and Esther Cheng and Jie Liu and Qunshu Lin and Raven Yuan and Tuney Zheng and Wei Pang and Xinrun Du and Yiming Liang and Yinghao Ma and Yizhi Li and Ziyang Ma and Bill Lin and Emmanouil Benetos and Huan Yang and Junting Zhou and Kaijing Ma and Minghao Liu and Morry Niu and Noah Wang and Quehry Que and Ruibo Liu and Sine Liu and Shawn Guo and Soren Gao and Wangchunshu Zhou and Xinyue Zhang and Yizhi Zhou and Yubo Wang and Yuelin Bai and Yuhan Zhang and Yuxiang Zhang and Zenith Wang and Zhenzhu Yang and Zijian Zhao and Jiajun Zhang and Wanli Ouyang and Wenhao Huang and Wenhu Chen},
    year    = {2024},
    journal = {arXiv preprint arXiv: 2405.19327}
}

数据概览

Matrix 面向大语言模型预训练，当前镜像主要包含 JSONL 语料和说明文件。数据规模很大，使用时应评估存储、带宽、去重质量和许可证覆盖范围。

496 个文件

浏览文件

数据集版权信息

本数据集的许可证为 Apache License 2.0。如有违反相关条款，请联系 WEHUB，我们将及时处理。查看许可证