New backbone LYNXNet2 replaces LYNXNet as the default for both acoustic and
variance models, using depthwise separable convolutions with multi-stage GLU
gating (SwiGLU or the new ATanGLU), a pre-norm design, and condition cache
support for ONNX export.
Muon optimizer (Momentum + Newton-Schulz orthogonalization) is introduced
along with ChainedOptimizer and Muon_AdamW for automatic parameter routing
between Muon (>=2D weights) and AdamW (1D params, embeddings, output layer).
Includes the Gram Newton-Schulz variant for ~42% speedup on rectangular
matrices and BF16 support.
Stretch embedding encodes phoneme stretching/compression ratio as a
conditioning signal for both acoustic and variance models.
Variance scaling normalizes all variance inputs (energy, breathiness, pitch,
etc.) to ~[-1, 1] ranges for improved training stability.
Other improvements: NeoX-style RoPE, improved weight initialization
(AdamWLinear, AdamWConv1d), DDP RoPE cache fix, and several ONNX export fixes.
Training configs updated: hidden_size 256->384, scheduler step_size 10000->5000,
smooth widths 0.12->0.06, duration predictor fs2->resnet, pitch/variance
backbone wavenet->lynxnet2.
**Breaking:** training resumption from old checkpoints is not supported due to
architecture changes. Old checkpoints still work for inference with their
bundled configs. Data does not need re-binarization.
---------
Co-authored-by: yxlllc <llc1995@sina.com>
Co-authored-by: Kakaru <97896816+KakaruHayate@users.noreply.github.com>
Co-authored-by: wolfgitpr <133209402+wolfgitpr@users.noreply.github.com>
yqzhishen
0482c1fbb3