-
[feat](kt-kernel): end-to-end full-parameter and LoRA SFT(#2094)
frostbyte_neo 发布于
2026-07-23 13:23:34 +00:00 | 34 次提交 在此版本后已推送到 main-
Add full FT development snapshot
-
fix: fix Full FT TP base weight gradients
-
[fix]: bug fix of 2d81e86
-
fix: fix AMX BF16 full-weight gradients
-
docs: document Full FT fork changes and debug history
-
docs: align fork remote terminology
-
chore: keep agent notes local
-
fix: configure SFT OpenMP threads
-
perf: optimize AMX Full-FT weight gradients
Coarsen base-weight gradient work from individual output tiles to fixed-intermediate strips so each task reuses packed panels across the hidden dimension.
Keep aligned thread-local BF16 panels across tasks and retain FP32 AMX accumulator tiles for the full K reduction. Gate and up run separate K passes while sharing the packed input panel.
On the matched Qwen3-30B-A3B 1-GPU test, stable Full-FT backward drops from 9.281s to 6.792s (-26.82%), step time drops from 19.252s to 16.140s, and TPS rises from 212.76 to 253.78. The LoRA-only backward control changes by -2.74%.
Validated with clang-format, the Release AMX/CUDA extension build, TP1/TP2 reference gradients across boundary token counts, and the 15-step Full-then-LoRA performance run.
-
feat: add staged SFT profiling
-
fix: reuse inference BF16 kernel for SFT
-
perf: add fine-grained Full-FT profiling
-
perf: batch BF16 Full-FT weight gradients
Use one expert-aggregated tile driver for AVX512-BF16 and AMX base-weight gradients, and pack updated full-precision weights directly into TP BufferB layouts without temporary partitions. Add worker-local profiling and focused dWeight/strided-repack coverage.
-
test: benchmark BF16 dWeight AMX driver
-
fix: label dWeight store as worker CPU time
-
docs: record BF16 Full-FT performance
-
docs: remove Qwen3 Full-FT performance report
-
perf: reduce BF16 Full-FT checkpoint overhead
Retain the first CPU MoE forward state across non-reentrant checkpoint recomputation, write BF16 activations directly into the backward cache, and reduce dWeight packing and gradient-clear traffic. Extend staged profiling and cover checkpoint reuse plus AMX/AVX dWeight paths.
- perf: make SFT optimizer gradients authoritative
Bind Full-FT and LoRA Parameter.grad directly to the KT-managed BF16 gradient buffers, avoiding PyTorch duplicate accumulation.
Accumulate microbatch gradients in C++, lazily clear expert buffers between optimizer windows, preserve rank-0 distributed ownership, and add lifecycle and AMX dWeight coverage.
- perf: avoid eager Full-FT gradient zeroing
Allocate authoritative Full-FT gradient buffers with torch.empty. The C++ state machine performs the mandatory full clear before first use, avoiding redundant Python-side first touch.
-
perf(sft): enable checkpoint forward reuse for LoRA
-
fix(sft): serialize checkpoint recompute with async repack
-
fix(sft): persist authoritative full weights
-
fix(sft): normalize legacy distributed gradients
-
fix: train gated shared experts
-
fix(sft): preserve expert placeholders across state dict loads
-
ci: publish ktransformers sdist in release workflow
-
fix(sft): preserve router autograd for LoRA training
Determine routing graph tracking from the router's trainable parameters instead of the Full-FT mode flag, so PEFT LoRA adapters on MoE gates receive routing-weight gradients. Fail fast when a trainable router returns detached weights and cover frozen, TopK, checkpoint-reuse, and two-step optimizer behavior.
Co-authored-by: illu wubowen03@foxmail.com
下载附件
-