githubnemo
daf335f503
The current state of the PEFT docs is not one of structure and I was constantly annoyed that whenever I wanted to change something there were several places that needed touching and they all felt disconnected. So this is my attempt at structuring the docs. Some of these ideas are quite old (discussed in 01/2025) but are still valid. I've removed most of the code guides without replacement. That's not ideal, I think we should have code examples but I'm think they should be method-focused. Maybe one general example of a training workflow is sufficient because most methods follow the same scheme. All details from the method guides (prompting, lora, oft/boft, etc.) are now integrated into the respective method pages instead. I would have hesitated to do this if these guides would have integrated information about the adapters but they didn't. I think it makes a lot more sense to have one place for each method to gather examples/tips/recommendations and that is now the `package_refernce/<method>` page. This page now also hosts a small space that shows the MetaMathQA (and potentially other) benchmark results highlighted for that method. I've moved the LoRA initializations to `package_reference/lora#Initialization` and converted the init methods to `<hfoption>`-tags. This collapses them to a list but may reduce searchability through the document - at least firefox is not able to search 'through' the option tabs. This also doesn't make them appear in the ToC and people specifically searching for, say, PiSSA won't find it directly. I think that's OK though, since the search is able to locate it. The quicktour is a bit more detailed about what happens under the hood (quick doesn't have to mean simplistic) and includes some new visualizations. I hope that we can integrate more visualizations in the future where it makes sense. * Remove PEFT method space + front page buttons The space was not that useful anymore since most methods are compatible with most models. The front page buttons are, at least temporarily, with the exception of the quicktour and method overview buttons. I like the visuals but there should only be elements that are useful. --------- Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
3.6 KiB
PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers
PEANuT is a parameter-efficient fine-tuning technique that introduces weight-aware neural tweakers to generate adapter updates from the frozen pretrained weights themselves. Instead of learning a purely linear low-rank update as in LoRA, PEANuT conditions the adapter transformation on the base weight, which makes the update rule more expressive while keeping the number of trainable parameters small.
PEANuT uses an input projection A, an output projection B, and optional intermediate residual encoder/decoder
pairs with non-linear activations. This makes it possible to model more complex update patterns than weight-agnostic
linear adapters while still remaining within the PEFT setting.
PEANuT currently has the following tradeoffs:
Pros:
- Higher theoretical expressiveness than linear low-rank updates.
- Better performance than LoRA on a range of tasks under similar budgets.
- Works well in very low-parameter regimes, for example around
0.2Mtrainable parameters.
Cons:
- Higher memory usage than LoRA, because
ΔWis explicitly constructed before being applied. - Slower training and inference than LoRA, and deeper intermediate layers increase the overhead further.
- The non-linearity can require more careful hyperparameter tuning, especially learning rate and related optimization settings.
If these tradeoffs do not fit your use case, consider other PEFT methods such as LoRA.
The abstract from the paper is:
Fine-tuning large pre-trained foundation models often yields excellent downstream performance but is prohibitively expensive when updating all parameters. Parameter-efficient fine-tuning (PEFT) methods such as LoRA alleviate this by introducing lightweight update modules, yet they commonly rely on weight-agnostic linear approximations, limiting their expressiveness. In this work, we propose PEANuT, a novel PEFT framework that introduces weight-aware neural tweakers, compact neural modules that generate task-adaptive updates conditioned on frozen pre-trained weights. PEANuT provides a flexible yet efficient way to capture complex update patterns without full model tuning. We theoretically show that PEANuT achieves equivalent or greater expressivity than existing linear PEFT methods with comparable or fewer parameters. Extensive experiments across four benchmarks with over twenty datasets demonstrate that PEANuT consistently outperforms strong baselines in both NLP and vision tasks, while maintaining low computational overhead.
Benchmark overview
API
PeanutConfig
autodoc tuners.peanut.config.PeanutConfig
PeanutModel
autodoc tuners.peanut.model.PeanutModel