github-featured / deepspeedai--deepspeed

发布

0 发布 128 Git 标签

RSS 订阅源
  • v0.13.2 18179807f5
    比较

    Remove optimizer step on initialization (#5104)

    frostbyte_neo 发布于 2024-02-11 01:47:45 +00:00

    All ZeRO 1/2/3 stages call the optimizer's step() on its
    initialization. This increments a counter in the optimizer and produces
    a different result in parameter update with the normal usage of PyTorch.
    This PR eliminates step() in the initialization and lazily configures
    some internal states (linking hp_params) after the first step()
    call.

    Co-authored-by: Olatunji Ruwase olruwase@microsoft.com

    下载附件