项目文件夹
Han Yu
29d0abbc21
## Summary - Route AutoEP's EP>1 DeepEP path directly from router/utilization accounting into `_deepep_route`, before collective-backend token preparation. - Avoid DeepEP-unused stable argsort, score/expert gathers, `[T*K,H]` routed-input expansion, score preparation, split-count all-to-all, and D2H split materialization. - Share the output finalization tail across DeepEP, standard communication, and EP1 so output shape, shared experts, router logits, and cache clearing retain their existing contracts. This is an opt-in backend cleanup: the default standard communication path and EP1 path are unchanged. Compatibility: - DeepEP continues to bypass fused weighted restore because its combine already restores and reduces token rows. - DeepEP now bypasses split-plan construction entirely, including async split planning. - The existing DeepEP incompatibility with `autoep_non_moe` compile is unchanged. - Folded tensor parallelism and standard communication retain their existing paths. ## Testing Done - [x] Local code review completed - [x] Unit tests added/updated - [x] Integration tests pass - [x] Manual testing performed Correctness: - Repository pre-commit hooks pass for all three changed files. - Rebased H100 targeted suite passed after upstream AutoEP score-correction bias changes. - CPU/mock contracts verify DeepEP does not call argsort, standard score application, or split-plan construction, while standard communication and EP1 still use their existing preparation. - H100 cleanup OFF/ON parity covers output, loss, exact routes, input gradient, routing-score and router-parameter gradients, all expert gradients, optimizer deltas, activation checkpointing on/off, skewed routing, and an empty expert. Performance, fixed-routing Qwen3-30B-A3B, EP16, TP1, BF16, activation checkpointing on, no profiler: - L8 smoke: `115.01 ms -> 109.05 ms`, a `5.18%` improvement. - L48 fresh ABBA + BAAB blocks: `682.86 ms -> 665.42 ms`, a `2.55%` improvement. - Four paired L48 deltas were all positive: `42.50`, `25.32`, `9.56`, and `18.56 ms`. - Paired mean delta was `23.98 ms`, with 95% CI `[1.82, 46.14] ms`. - Maximum loss difference was `0.00682`; p95 did not regress. - Peak allocated/reserved changed by approximately `+2.2 MiB / +22 MiB`, within the no-regression gate. Profiling: - Sparse CUDA-event observer overhead was `-1.69%`, within run-to-run noise. - The early-route path emits no standard split-plan preparation. - One steady-state lean Nsight capture was collected only for explanation; traced wall time is not used as the performance headline. Matched DeepEP V2 context (same-allocation paired blocks; not a merge gate): - Megatron uses NVIDIA/Megatron-LM#5153 head `eb688c4a...` with the same DeepEP `01dc3aaa...`, ElasticBuffer V2, NCCL 2.30.4, 12 SMs, 16 QPs, fixed routing, data, and checkpointing semantics. - Two fresh allocations each discarded one full AutoEP arm and one full Megatron arm before ABBA/BAAB measurement. - All four paired L48 deltas favored AutoEP: `38.62`, `67.21`, `80.91`, and `62.18 ms`. - Paired mean delta was `62.23 ms`, with 95% CI `[34.20, 90.26] ms`; pooled medians were `649.76 ms` for AutoEP and `714.45 ms` for Megatron, an AutoEP speedup of `9.06%`. - Measured-window loss differed by at most `0.00593`. - AutoEP used about `0.63 GiB` more allocated and `2.77 GiB` more reserved memory. - The median result does not represent average elapsed time: across all 80 recorded measured steps, AutoEP averaged `766.67 ms` versus `716.22 ms` for Megatron, so AutoEP was about `7.0%` slower by arithmetic mean. - AutoEP had `32/80` steps over `800 ms` versus `0/80` for Megatron; only `3/80` crossed one second. The repeated slow steps dominate full-run throughput and remain unexplained. Follow-up fixed-routing instrumentation localizes those repeated slow steps: - Two additional AutoEP arms again had fast medians (`633.17` and `638.05 ms`) but slower means (`754.56` and `747.22 ms`), with `16/40` measured steps over `800 ms`. - Slow versus fast median inflation was concentrated in forward (`+281 ms` and `+248 ms` in the two arms); backward and optimizer medians were effectively unchanged. - A detailed follow-up arm attributed essentially all of the slow-forward increase to the DeepEP dispatch call: `+382.07 ms` dispatch versus `+1.12 ms` expert compute, `-3.43 ms` combine, and `+3.94 ms` router. - Cross-rank inspection shows that dispatch is the synchronization surface, not yet the root cause: at every hotspot, `14-15` ranks wait about `233-357 ms`, while one late-arriving rank spends only about `0.55-0.74 ms` in dispatch. The late rank's preceding MoE/combine work is normally only `1.5-3.0 ms`, placing most of the originating delay in the uninstrumented non-MoE forward region between MoE layers; one observed case accumulated the delay in the router call itself. - The synchronized measured-window wallclock was `16893.88 ms`, versus `16894.84 ms` from summing the 20 recorded critical-step times. The step measurements therefore account for the full measured training window; the mean/median reversal is not an omitted gap between steps. - A second detailed run split the inter-MoE region into decoder, attention, and normalization calls. The delayed call site moved between input RMSNorm, self-attention, router, and otherwise uninstrumented Python gaps on different ranks and layers. In each case one rank paused for roughly `260-435 ms`, after which the remaining ranks waited in the next dispatch. This pattern is inconsistent with a specific DeepEP or transformer kernel regression. - A causal run with identical fixed routing and instrumentation but Python cyclic GC disabled after initialization removed the tail completely: mean `933.08 -> 557.31 ms`, p95 `1740.38 -> 581.60 ms`, and steps over `800 ms` `11/20 -> 0/20`. The measured-window mean was `557.24 ms`, matching the recorded `557.31 ms`. This comparison used separate allocations, so a same-allocation paired confirmation is still required before treating the magnitude as final. The same-allocation AutoEP-only confirmation is now complete. A dual-warm `default -> managed -> managed -> default` block used full measured-window timing: - Default automatic GC: `688.57 ms` mean, `535.13 ms` median, `1072.52 ms` median p95, `11/40` steps over `800 ms`, and `4/40` over one second. - `python_gc_policy="disable_during_training"`: `526.42 ms` mean, `522.89 ms` median, `551.12 ms` median p95, and no steps over `800 ms`. - Paired measured-window savings were `166.37` and `157.92 ms`; the paired mean was `162.14 ms`, with 95% CI `[108.45, 215.84] ms`. - Peak allocated/reserved memory was identical, observed routing was identical, and maximum paired loss differences were `0.00266` and `0.01439`. The opt-in engine-managed policy is isolated in Draft PR #8451. It is independent of this PR's DeepEP local-preparation cleanup. Rewriting or overlapping DeepEP dispatch would optimize the waiting point rather than the source of the tail. Natural-routing same-allocation context: - All four paired L48 deltas also favored AutoEP: `67.03`, `55.87`, `80.45`, and `101.26 ms`. - Paired mean delta was `76.15 ms`, with 95% CI `[45.10, 107.21] ms`; pooled medians were `686.49 ms` for AutoEP and `760.23 ms` for Megatron, an AutoEP speedup of `9.70%`. - Measured-window loss differed by at most `0.00970`. - Across all 80 measured steps, AutoEP averaged `793.65 ms` versus `801.08 ms` for Megatron, only a `0.93%` average-time advantage despite the larger median signal. - AutoEP had `28/80` steps over `800 ms` versus `14/80` for Megatron, and `9/80` over one second versus `2/80`; only one AutoEP step exceeded twice its arm median. - A separate tokens-per-expert audit showed that the natural-routing workload is not fully matched. Per-layer sorted expert-load total variation had a `9.84%` median, but indexed expert variation had a `64.72%` median and rank-receive variation had a `46.46%` median. The two frameworks therefore see similar load-shape distributions assigned to different expert/rank identities. - The natural-routing speedup is reported as end-to-end context, not as a pure framework execution gap. The fixed-routing paired comparison remains the controlled cross-framework result. 🤖 Generated with [GitHub Copilot CLI](https://docs.github.com/copilot/github-copilot-cli) --------- Signed-off-by: yh0903 <helloyu0903@gmail.com> Signed-off-by: Masahiro Tanaka <tanaka.masahiro@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com> Co-authored-by: Masahiro Tanaka <tanaka.masahiro@gmail.com>
Office Hours
DeepSpeed hosts regular office hours on the last Tuesday of each month at 12:00 America/New_York to discuss development plans, features, etc. This meeting is public for anyone to join and ask questions. The meeting is hosted on Zoom and can be joined here.
Latest News
-
[2026/05] Using Muon Optimizer with DeepSpeed
-
[2026/05] System DMA (SDMA) for ZeRO-3: offload collectives off compute units on AMD GPUs for better overlap
-
[2026/03] DeepSpeed Team gave a tutorial at ASPLOS 2026 titled "Building Efficient Large-Scale Model Systems with DeepSpeed: From Open-Source Foundations to Emerging Research"
-
[2026/03] Our SuperOffload work received an Honorable Mention for the ASPLOS 2026 Best Paper Award
-
[2025/12] DeepSpeed Core API updates: PyTorch-style backward and low-precision master states
-
[2025/10] We hosted the Ray x DeepSpeed Meetup at Anyscale. We shared our most recent work on SuperOffload, ZenFlow, Muon Optimizer Support, Arctic Long Sequence Training and DeepCompile. Please find the meetup slides here.
-
[2025/10] SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips
-
[2025/10] Study of ZenFlow and ZeRO offload performance with DeepSpeed CPU core binding
-
[2025/08] ZenFlow: Stall-Free Offloading Engine for LLM Training
-
[2025/06] DeepNVMe: Affordable I/O scaling for Deep Learning Applications
More news
Extreme Speed and Scale for DL Training
DeepSpeed enabled the world's most powerful language models (at the time of this writing) such as MT-530B and BLOOM. DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. These innovations include ZeRO, ZeRO-Infinity, 3D-Parallelism, Ulysses Sequence Parallelism, DeepSpeed-MoE, etc.
DeepSpeed Adoption
DeepSpeed was an important part of Microsoft’s AI at Scale initiative to enable next-generation AI capabilities at scale, where you can find more information here.
DeepSpeed has been used to train many different large-scale models, below is a list of several examples that we are aware of (if you'd like to include your model please submit a PR):
- Megatron-Turing NLG (530B)
- Jurassic-1 (178B)
- BLOOM (176B)
- GLM (130B)
- xTrimoPGLM (100B)
- YaLM (100B)
- GPT-NeoX (20B)
- AlexaTM (20B)
- Turing NLG (17B)
- METRO-LM (5.4B)
DeepSpeed has been integrated with several different popular open-source DL frameworks such as:
| Documentation | |
|---|---|
  |
Transformers with DeepSpeed |
  |
Accelerate with DeepSpeed |
  |
Lightning with DeepSpeed |
 |
MosaicML with DeepSpeed |
 |
Determined with DeepSpeed |
 |
MMEngine with DeepSpeed |
Build Pipeline Status
| Description | Status |
|---|---|
| NVIDIA |   |
| AMD |  |
| CPU |  |
| Intel Gaudi |  |
| Intel XPU |  |
| Integrations |  |
| Misc |    |
| Huawei Ascend NPU |  |
Installation
The quickest way to get started with DeepSpeed is via pip, this will install the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA versions. DeepSpeed includes several C++/CUDA extensions that we commonly refer to as our 'ops'. By default, all of these extensions/ops will be built just-in-time (JIT) using torch's JIT C++ extension loader that relies on ninja to build and dynamically link them at runtime.
Requirements
- PyTorch must be installed before installing DeepSpeed.
- For full feature support we recommend a version of PyTorch that is >= 2.0 and ideally the latest PyTorch stable release.
- A CUDA or ROCm compiler such as nvcc or hipcc used to compile C++/CUDA/HIP extensions.
- Specific GPUs we develop and test against are listed below, this doesn't mean your GPU will not work if it doesn't fall into this category it's just DeepSpeed is most well tested on the following:
- NVIDIA: Pascal, Volta, Ampere, and Hopper architectures
- AMD: MI100 and MI200
Contributed HW support
- DeepSpeed now support various HW accelerators.
| Contributor | Hardware | Accelerator Name | Contributor validated | Upstream validated |
|---|---|---|---|---|
| Huawei | Huawei Ascend NPU | npu | Yes | No |
| Intel | Intel(R) Gaudi(R) 2 AI accelerator | hpu | Yes | Yes |
| Intel | Intel(R) Xeon(R) Processors | cpu | Yes | Yes |
| Intel | Intel(R) Data Center GPU Max series | xpu | Yes | Yes |
| Tecorigin | Scalable Data Analytics Accelerator | sdaa | Yes | No |
PyPI
We regularly push releases to PyPI and encourage users to install from there in most cases.
pip install deepspeed
After installation, you can validate your install and see which extensions/ops your machine is compatible with via the DeepSpeed environment report.
ds_report
If you would like to pre-install any of the DeepSpeed extensions/ops (instead of JIT compiling) or install pre-compiled ops via PyPI please see our advanced installation instructions.
Windows
Many DeepSpeed features are supported on Windows for both training and inference. You can read more about this in the original blog post here. Among features that are currently not supported are async io (AIO) and GDS (which does not support Windows).
- Install PyTorch, such as pytorch 2.3+cu121.
- Install Visual C++ build tools, such as VS2022 C++ x64/x86 build tools.
- Launch Cmd console with Administrator permissions for creating required symlink folders and ensure MSVC tools are added to your PATH or launch the Developer Command Prompt for Visual Studio 2022 with administrator permissions.
- Run
build_win.batto build wheel indistfolder.
Further Reading
All DeepSpeed documentation, tutorials, and blogs can be found on our website: deepspeed.ai
| Description | |
|---|---|
| Getting Started | First steps with DeepSpeed |
| DeepSpeed JSON Configuration | Configuring DeepSpeed |
| API Documentation | Generated DeepSpeed API documentation |
| Tutorials | Tutorials |
| Blogs | Blogs |
CI funding
This being an open source project we rely on others to provide us resources for CI hardware. At this moment Modal is kindly supporting our GPU CI runs by funding the hardware for us. Modal is an AI infrastructure platform for inference, fine-tuning, batch jobs and more. Get started with $30/mo in free credits today at https://modal.com. We have been getting an amazing support from Modal's team and will surely recommend them to your business.
Contributing
DeepSpeed welcomes your contributions! Please see our
contributing guide for more details on formatting, testing,
etc.
Thanks so much to all of our amazing contributors!
Developer Certificate of Origin
This project welcomes contributions and suggestions. Most contributions require you to agree to a Developer Certificate of Origin DCO stating that they agree to the terms published at https://developercertificate.org for that particular contribution.
DCOs are per-commit, so each commit needs to be signed off. These can be signed in
the commit by adding the -s flag. DCO enforcement can also be signed off in the PR
itself by clicking on the DCO enforcement check.
Code of Conduct
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
Publications
-
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He. (2019) ZeRO: memory optimizations toward training trillion parameter models. arXiv:1910.02054 and In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20).
-
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. (2020) DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial).
-
Minjia Zhang, Yuxiong He. (2020) Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping. arXiv:2010.13369 and NeurIPS 2020.
-
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He. (2021) ZeRO-Offload: Democratizing Billion-Scale Model Training. arXiv:2101.06840 and USENIX ATC 2021. [paper] [slides] [blog]
-
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He. (2021) 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. arXiv:2102.02888 and ICML 2021.
-
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He. (2021) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. arXiv:2104.07857 and SC 2021. [paper] [slides] [blog]
-
Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He. (2021) 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed. arXiv:2104.06069 and HiPC 2022.
-
Conglong Li, Minjia Zhang, Yuxiong He. (2021) The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models. arXiv:2108.06084 and NeurIPS 2022.
-
Yucheng Lu, Conglong Li, Minjia Zhang, Christopher De Sa, Yuxiong He. (2022) Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam. arXiv:2202.06009.
-
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He. (2022) DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale arXiv:2201.05596 and ICML 2022. [pdf] [slides] [blog]
-
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro. (2022) Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model arXiv:2201.11990.
-
Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong He. (2022) Extreme Compression for Pre-trained Transformers Made Simple and Efficient. arXiv:2206.01859 and NeurIPS 2022.
-
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He. (2022) ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. arXiv:2206.01861 and NeurIPS 2022 [slides] [blog]
-
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He. (2022) DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. arXiv:2207.00032 and SC 2022. [paper] [slides] [blog]
-
Zhewei Yao, Xiaoxia Wu, Conglong Li, Connor Holmes, Minjia Zhang, Cheng Li, Yuxiong He. (2022) Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers. arXiv:2211.11586.
-
Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Yuxiong He. (2022) DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing. arXiv:2212.03597 ENLSP2023 Workshop at NeurIPS2023
-
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He. (2023) Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases. arXiv:2301.12017 and ICML2023.
-
Syed Zawad, Cheng Li, Zhewei Yao, Elton Zheng, Yuxiong He, Feng Yan. (2023) DySR: Adaptive Super-Resolution via Algorithm and System Co-design. ICLR:2023.
-
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, Yuxiong He. (2023) Scaling Vision-Language Models with Sparse Mixture of Experts. arXiv:2303.07226 and Finding at EMNLP2023.
-
Quentin Anthony, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He, Aamir Shafi, Mustafa Abduljabbar, Hari Subramoni, Dhabaleswar Panda. (2023) MCR-DL: Mix-and-Match Communication Runtime for Deep Learning arXiv:2303.08374 and will appear at IPDPS 2023.
-
Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele. (2023) A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training arXiv:2303.06318 and ICS 2023.
-
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He. (2023) ZeRO++: Extremely Efficient Collective Communication for Giant Model Training arXiv:2306.10209 and ML for Sys Workshop at NeurIPS2023 [blog]
-
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, Yuxiong He. (2023) ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation arXiv:2303.08302 and ENLSP2023 Workshop at NeurIPS2023 [slides]
-
Pareesa Ameneh Golnari, Zhewei Yao, Yuxiong He. (2023) Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important? arXiv:2305.09847
-
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, Yuxiong He. (2023) DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales arXiv:2308.01320.
-
Xiaoxia Wu, Zhewei Yao, Yuxiong He. (2023) ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats arXiv:2307.09782 and ENLSP2023 Workshop at NeurIPS2023 [slides]
-
Zhewei Yao, Xiaoxia Wu, Conglong Li, Minjia Zhang, Heyang Qin, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He. (2023) DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention arXiv:2309.14327
-
Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang, Conglong Li, Shiyang Chen, Chengming Zhang, Masahiro Tanaka, Xiaoxia Wu, Jeff Rasley, Ammar Ahmad Awan, Connor Holmes, Martin Cai, Adam Ghanem, Zhongzhu Zhou, Yuxiong He, et al. (2023) DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies arXiv:2310.04610 [blog]
-
Zhewei Yao, Reza Yazdani Aminabadi, Stephen Youn, Xiaoxia Wu, Elton Zheng, Yuxiong He. (2023) ZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 Transformers arXiv:2310.17723
-
Xiaoxia Wu, Haojun Xia, Stephen Youn, Zhen Zheng, Shiyang Chen, Arash Bakhtiari, Michael Wyatt, Reza Yazdani Aminabadi, Yuxiong He, Olatunji Ruwase, Leon Song, Zhewei Yao (2023) ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks arXiv:2312.08583
-
Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song. (2024) FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design arXiv:2401.14112
-
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminadabi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He. (2024) System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
-
Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, Minjia Zhang. (2024) Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training arXiv:2406.18820
-
Stas Bekman, Samyam Rajbhandari, Michael Wyatt, Jeff Rasley, Tunji Ruwase, Zhewei Yao, Aurick Qiao, Yuxiong He. (2025) Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences arXiv:2506.13996
-
Tingfeng Lan, Yusen Wu, Bin Ma, Zhaoyuan Su, Rui Yang, Tekin Bicer, Masahiro Tanaka, Olatunji Ruwase, Dong Li, Yue Cheng. (2025) ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates arXiv:2505.12242
-
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder.(2025) Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems EMNLP 2025
-
Xinyu Lian, Masahiro Tanaka, Olatunji Ruwase, Minjia Zhang. (2026) SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips arxiv, ASPLOS 2026
Videos
- DeepSpeed KDD 2020 Tutorial
- Overview
- ZeRO + large model training
- 17B T-NLG demo
- Fastest BERT training + RScan tuning
- DeepSpeed hands on deep dive: part 1, part 2, part 3
- FAQ
- Microsoft Research Webinar
- Registration is free and all videos are available on-demand.
- ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed.
- DeepSpeed on AzureML
- Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference [slides]
- Community Tutorials