-
Improving memory utilization of Z2+MoE (#2079)
frostbyte_neo 发布于
2022-07-13 23:02:02 +00:00 - Shards expert parameter groups
- Do upscaling, optimizer and deletion of fp32 grads one-by-one on each parameter group in zero-2
Co-authored-by: Olatunji Ruwase olruwase@microsoft.com
下载附件