github-featured / deepspeedai--deepspeed

发布

0 发布 128 Git 标签

RSS 订阅源
  • v0.16.3 c17dc33c04
    比较

    Using explicit GPU upcast for ZeRO-Offload (#6962)

    frostbyte_neo 发布于 2025-01-21 18:48:38 +00:00

    Following discussion in
    PR-6670, the explict
    upcast is much more efficient than implicit upcast, this PR is to
    replace implicit upcast with explict one.

    The results on 3B model are shown below:

    Option BWD (ms) Speed up
    Before PR-6670 25603.30 1x
    After PR-6670 1174.31 21.8X
    After this PR 309.2 82.8X
    下载附件