Oswaldo Ludwig
14e0a59a41
Implements KappaTune based on: "The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units" https://arxiv.org/abs/2506.16289 This is a helper function that can be run on the base model to identify the layers that are most suited for fine-tuning. It works for Linear and MoE layers. By targeting these layers with LoRA or other PEFT methods, the model should train well without forgetting useful information from pre-training.
1.4 KiB
Helper methods
A collection of helper functions for PEFT.
Checking if a model is a PEFT model
autodoc helpers.check_if_peft_model - all
Temporarily Rescaling Adapter Scale in LoraLayer Modules
autodoc helpers.rescale_adapter_scale - all
Context manager to disable input dtype casting in the forward method of LoRA layers
autodoc helpers.disable_input_dtype_casting - all
Context manager to enable DoRA caching (faster at inference time but requires more memory)
autodoc helpers.DoraCaching - all
KappaTune target selection
KappaTuneSelector and find_kappa_target_modules implement a general target selection process from the KappaTune paper.
The method identifies modules with higher flexibility (higher output differential entropy) and lower specialization (lower sensitivity to specific input directions).
These properties make the selected modules good candidates for mitigating catastrophic forgetting in any adaptation method that adds trainable parameters, including LoRA, DoRA, LoHa, AdaLoRA, and even direct fine-tuning of the original weights.
autodoc helpers.KappaTuneSelector - all
autodoc helpers.find_kappa_target_modules - all