数据集 / inclusionAI/LLaDA2.0-flash-preview

inclusionAI/LLaDA2.0-flash-preview 已完整同步

LLaDA2.0-flash-preview

LLaDA2.0-flash-preview is a diffusion language model featuring a 100BA6B Mixture-of-Experts (MoE) architecture. As an enhanced, instruction-tuned iteration of the LLaDA2.0 series, it is optimized for practical applications.

Benchmark Ling-flash-2.0 LLaDA2.0-mini-preview LLaDA2.0-flash-preview
Average 79.93 66.89 77.03
Knowledge
MMLU 87.98 72.49 83.15
MMLU-PRO 76.84 49.22 66.16
CMMLU 86.59 67.53 79.64
C-EVAL 88.03 66.54 79.28
Reasoning
squad2.0 81.32 85.61 90.61
drop 88.32 79.49 88.17
korbench 68.96 37.26 53.28
Coding
CruxEval-O 82.75 61.88 74.50
mbpp 85.01 77.75 86.65
MultiPL-E 65.76 62.43 72.38
humaneval 85.98 80.49 88.41
Bigcodebench-Full 40.70 30.44 40.44
Math
GSM8K 95.45 89.01 95.75
math 96.1 73.50 83.52
Agent & Alignment
BFCL_Live 67.57 74.11 74.86
IFEval-strict -prompt 81.52 62.50 75.60

🚀 Performance Highlights

  • Leading MoE Architecture: The open-source Mixture-of-Experts (MoE) diffusion large language model continually trained on the Ling2.0 series with approximately 20 trillion tokens.
  • Efficient Inference: With 100 billion total parameters, only 6.1 billion are activated during inference. LLaDA2.0-flash-preview significantly reduces computational costs while outperforming open-source dense models of similar scale.
  • Impressive Performance on Code & Complex Reasoning: Excels in tasks such as code generation and advanced mathematical reasoning, demonstrating strong reasoning capabilities.
  • Tool Use: Supports tool calling and achieves excellent performance in complex agent-based tasks.
  • Open & Extensible: Fully open-source with commitment to transparency. We plan to release a leading inference framework in the future and continue investing in cutting-edge areas like diffusion LLMs (dLLM) to drive disruptive innovation.

🗺️ What's Next

  • Supercharged Reasoning with LLaDA 2.0: LLaDA 2.0 series will be fine-tuned with Reinforcement Learning, unlocking a new level of sophisticated reasoning and problem-solving abilities.
  • Tools for Innovators: The model was finetuned on the VeOmni framework using Fully Sharded Data Parallel (FSDP2). We will release a detailed tutorial and our complete post-training framework. Whether you want to master the current model or build your own customized versions, you'll have the tools you need. Stay tuned

📦 Model Variants

Model ID Description Hugging Face Link
inclusionAI/LLaDA2.0-mini-preview Instruction-tuned model, ready for downstream applications. 🤗 Model Card
inclusionAI/LLaDA2.0-flash-preview Instruction-tuned model, ready for downstream applications. 🤗 Model Card

🔍 Model Overview

LLaDA2.0-flash-preview has the following specifications:

  • Type: Mixture-of-Experts (MoE) Diffusion Language Model
  • Total Parameters (Non-Embedding): 100B
  • Number of Layers: 32
  • Attention Heads: 32
  • Context Length: 4,096 tokens
  • Position Embedding: Rotary (RoPE)
  • Vocabulary Size: 157,184

🤗 Hugging Face Transformers

Make sure you have transformers and its dependencies installed:

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausa

52 个文件

浏览文件

数据集版权信息

本数据集的许可证为 Apache License 2.0。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证