Resolves#3340
Context
So far, we did not allow adding multiple LoRA adapters with
target_parameters on the same layer. This was a known limitation. I
have already attempted to solve this once (see #2710) but didn't have
time to come up with a nice solution. As it was unclear if there was
any real world need to support this, there was no further work on it
since then. Now we know that there are practical application that may
need it, so I resumed the work.
What doesn't work
The previous solution attempted to solve the issue by nesting the
lora.ParamWrappers. So for adapters 'default' and 'other', we would
end up having something like:
param_wrapper_default(param_wrapper_other(base_layer))
This was problematic. Not only could this result in very deep nesting,
which is inefficient. What's worse is that state_dict key for 'other'
would contain 'base_layer.' as an infix. Therefore, if we wanted to
load the 'other' adapter _without_ first loading the 'default'
adapter, we would get a key mismatch.
We could also not simply strip out 'base_layer.' infix because we use
nesting to deal with multiple nn.Parameters on the same module, so to
account for that, we need to keep the infix.
Solution
The solution is pretty straightfoward: We use the existing mechanism
to store the parameters for the other adapter in the
nn.ModuleDict. For this, we detect if the layer is already a
ParamWrapper when adding the second adapter and update that layer
instead of nesting it.
Caveat
This simple approach can, however, not work with multiple adapters
that target a different set of parameters. This is because the
information which parameter is targeted is not stored in the
state_dict itself. Therefore, if we had different adapters targeting
different parameters, we would not be able to tell which parameter is
meant to be targeted.
Adds DEFT: "DEFT: Decompositional Efficient Fine-Tuning for
Text-to-Image Models"
https://arxiv.org/abs/2509.22793
DEFT splits a weight update into two learned low-rank parts: a
projection that removes a sub-space of the frozen weight, and a low-rank
update that injects new content in its place. This allows DEFT to adapt
new data or concepts, e.g. personalizing a text-to-image model from a
few images, while retaining the base model's
instruction-following/editability with minimal forgetting. It's less
suitable if you don't need to preserve the base model's other
capabilities or for layers beyond Linear/Conv1D.
PaRa ("Personalizing Text-to-Image Diffusion via Parameter Rank
Reduction") is also supported as a special case by passing para=True.
https://arxiv.org/abs/2406.05641
In the doc navigation, under "LoRA > Initialization", there is a
"Usage Tips" section but clicking it doesn't work. The reason is that
it's a subsection of LoRA-GA, but LoRA-GA is not open by default as
the initialization options are formatted as different tabs.
My proposal is to remove the section title so that it no longer
appears in the navigation.
## Motivation
This PR adds **UniLoRA**, a LoRA-style parameter-efficient fine-tuning method
that introduces a unified parameterization for low-rank adaptations, enabling
further reductions in the number of trainable parameters while preserving
the standard PEFT workflow.
## What's included
- UniLoRA tuner implementation
- Configuration class and registry integration
- Save/load support
- Unit tests
## Blurb for publication
UniLoRA shares a compact trainable vector bank across low-rank adapter weights. It keeps the familiar PEFT training flow while using deterministic projections into shared `theta_d` values to reduce the number of trained adapter parameters.
Adds GLoRA: "One-for-All: Generalized LoRA for Parameter-Efficient
Fine-tuning"
https://huggingface.co/papers/2306.07967
GLoRA (Generalized LoRA) is a flexible PEFT method that extends LoRA
with configurable weight, activation, and bias adaptation, delivering
richer fine-tuning with no extra inference cost. Use it when you need
per-layer flexibility or stronger adaptation than vanilla LoRA. Skip it
for non-Linear layers (e.g. Conv/Embedding) or when standard LoRA is
already sufficient and simplicity matters.
Add: MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning
(https://arxiv.org/abs/2604.01694)
MiCA (Minor Component Adaptation) is a new initialization option for
LoRA. It initializes the LoRA B weight to the minor components of the
base weight and then freezes it, only learning A. This promises to
improve knowledge acquisition while reducing catastrophic forgetting and
reducing the number of parameters that need to be trained.
The question recently came up if hotswapping works with
target_parameters. Therefore, I added a test to check it. It turns out
that it works indeed.
The usefulness is, however, somewhat reduced because targeting
parameters while using torch.compile (compiled models are a main use
case for hotswapping) leads to re-compilation and/or graph
breaks. This is a fundamental limitation of how targeting
nn.Parameters is implemented, using nn.utils.parametrize to
dynamically update the targeted nn.Parameter. We can't update it
statically, since that would break all kinds of things (e.g. accessing
the parameter with model.foo.bar would return the parameter *after*
applying the LoRA delta weight). Therefore, we must undo the
parametrization after the forward step, and this breaks compilation.
This PR additionally documents the fundamental problem with
torch.compile and target_parameters. It also removes an unused
argument in a test and an incorrect comment.
Implements: FRoD: Full-Rank Efficient Fine-Tuning with Rotational
Degrees for Fast Convergence
https://arxiv.org/abs/2512.23485
FRoD is a full-rank, replacement-style PEFT method from FRoD: Full-Rank
Efficient Fine-Tuning with Rotational Degrees for Fast
Convergence. Instead of adding low-rank deltas, it reconstructs selected
weights with shared rotational subspaces and sparse trainable
coefficients. It is especially useful when fast convergence and a higher
full-rank capacity ceiling are important, and its large sparse
rotational subspace may also be promising for model merging. The main
tradeoffs are the costly joint-decomposition initialization and slightly
slower forward/backward passes than LoRA due to the sparse structured
factors, so it may be less attractive for a one-off single-task
fine-tune.
The current state of the PEFT docs is not one of structure and I was constantly annoyed that whenever I wanted to change something there were several places that needed touching and they all felt disconnected. So this is my attempt at structuring the docs. Some of these ideas are quite old (discussed in 01/2025) but are still valid.
I've removed most of the code guides without replacement. That's not ideal, I think we should have code examples but I'm think they should be method-focused. Maybe one general example of a training workflow is sufficient because most methods follow the same scheme.
All details from the method guides (prompting, lora, oft/boft, etc.) are now integrated into the respective method pages instead. I would have hesitated to do this if these guides would have integrated information about the adapters but they didn't. I think it makes a lot more sense to have one place for each method to gather examples/tips/recommendations and that is now the `package_refernce/<method>` page. This page now also hosts a small space that shows the MetaMathQA (and potentially other) benchmark results highlighted for that method.
I've moved the LoRA initializations to `package_reference/lora#Initialization` and converted the init methods to `<hfoption>`-tags. This collapses them to a list but may reduce searchability through the document - at least firefox is not able to search 'through' the option tabs. This also doesn't make them appear in the ToC and people specifically searching for, say, PiSSA won't find it directly. I think that's OK though, since the search is able to locate it.
The quicktour is a bit more detailed about what happens under the hood (quick doesn't have to mean simplistic) and includes some new visualizations. I hope that we can integrate more visualizations in the future where it makes sense.
* Remove PEFT method space + front page buttons
The space was not that useful anymore since most methods are compatible
with most models.
The front page buttons are, at least temporarily, with the exception
of the quicktour and method overview buttons. I like the visuals
but there should only be elements that are useful.
---------
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Implements KappaTune based on:
"The Condition Number as a Scale-Invariant Proxy for Information Encoding in
Neural Units"
https://arxiv.org/abs/2506.16289
This is a helper function that can be run on the base model to identify the
layers that are most suited for fine-tuning. It works for Linear and MoE
layers. By targeting these layers with LoRA or other PEFT methods, the model
should train well without forgetting useful information from pre-training.
Adds "HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large
Language Models" (https://openreview.net/pdf?id=TwJrTz9cRS)
This PEFT method is similar to LoRA but instead of updates to the base
weights being additive, they are multiplicative here (Hadamard product).
This promises to resolve some limitations of the low rank updates
provided by LoRA, especially for learning tasks requiring high
expressivity. Convolutional and embedding layers, as well as
bitsandbytes quantization, are supported.
Adds: "BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data
Regimes"
Paper: https://arxiv.org/abs/2509.15974
BEFT only learns the bias terms of the targeted layers, making it
extremely parameter efficient. Results show that BEFT works especially
well when targeting the V projection in low data regimes. When there is
a lot of training data and higher learning capacity is required, BEFT is
not the best fit.
While optimizing the hyper-parameters for prefix tuning in the MetaMathQA benchmark
most of the results turned out to have ~0% task accuracy (except for the baseline
configuration). Initializing the prefixes to be a no-op in the beginning, similar to
LoRA's default initialization, turned out to resolve this:
* Baseline: ~20% task accuracy
* Zero-init: ~35% task accuracy
The benchmark setup did not allow for passing an KV cache initialization string
which this change also corrects. Initializing the KV cache with a simple string
like "Question:" reaches a similar task accuracy (~36%) while having a lot less
forgetting than zero init.
A longer, task specific sequence (50 tokens) reaches ~43% task accuracy and
~0.5 forgetting.
Implements "AdaMSS: Adaptive Multi-Subspace Approach for
Parameter-Efficient Fine-Tuning"
https://openreview.net/forum?id=8ZdWmpYxT0
AdaMSS segments the base weights of the model into smaller subspaces
that are targeted for fine-tuning. Moreover, it's possible to
dynamically assign a lower parameter budget to less important subspaces
during training, similar to what AdaLoRA does. This promises to provide
higher expressiveness and better generalization than similar PEFT
methods.
Adds TinyLoRA, a new PEFT method based on "TinyLoRA: Learning to Reason in 13 Parameters". TinyLoRA achieves extreme parameter efficiency by replacing LoRA's trainable low-rank matrices with a tiny trainable vector projected through fixed random bases.
The key idea: given a frozen SVD decomposition `W ≈ B @ A` (where `B = U @ sqrt(S)` and `A = sqrt(S) @ V^T`), the weight update is `delta_W = B @ R @ A` where `R` is an `r x r` trainable matrix (following LoRA-XS). TinyLoRA takes this further by parameterizing `R` as a linear combination of fixed random projection matrices:
R = sum_i(v[i] * P[i])
where `v` is the only trainable parameter (as small as 13 values) and `P_i` are fixed random matrices seeded deterministically.
## Features
- Extreme efficiency: trainable parameter count is `u` per target module (or even less with weight tying), compared to `r * (in + out)` for LoRA
- Weight tying: configurable sharing of `v` vectors across layers via `weight_tying` (0.0 = no sharing, 1.0 = all layers share one `v`)
- SVD initialization: frozen `A` and `B` matrices computed from truncated SVD of pretrained weights, with singular values distributed equally via `sqrt(S)`
- Full layer support: `nn.Linear`, `Conv1D`, and `nn.Embedding`
- Merge/unmerge: full support including safe merge with NaN checking
- LoRA conversion: `supports_lora_conversion()` -> True — delta weights can be converted to standard LoRA format via `get_delta_weight`
- Deterministic projections: `P` matrices are seeded per-layer for reproducibility; optionally saved in checkpoints (`save_projection=True`)
## Config
```
from peft import TinyLoraConfig, get_peft_model
config = TinyLoraConfig(
r=2, # SVD rank (frozen)
u=64, # trainable vector dimension
weight_tying=0.0, # 0.0=no sharing, 1.0=full sharing
target_modules="all-linear",
)
model = get_peft_model(base_model, config)
```
## Architecture
- TinyLoraLayer (base): SVD decomposition, projection init, `get_delta_weight`, `supports_lora_conversion`
- Linear / Embedding: forward pass, merge/unmerge
- TinyLoraModel: weight tying groups, shared v parameter management via nested ModuleDict/ParameterDict
- update_layer follows LoRA's config-object pattern: (adapter_name, tinylora_v, v_key, r, config, **kwargs)
---------
Co-authored-by: githubnemo <githubnemo@users.noreply.github.com>
DeloRA is missing some config validation that other tuners already have (exclude_modules list-to-set conversion, r validation, Literal type for bias). This PR fixes that.
Also fixes some typos I noticed while going through the code.
Add PEANut: Parameter-Efficient Adaptation with Weight-aware Neural
Tweakers
Paper: https://arxiv.org/abs/2410.01870
PEANuT adds a small neural net (weight-aware neural tweakers). Compared
to LoRA, this increases expressivity for the same trainable parameter
count or allows to greatly lower the parameter count without sacrificing
expressivity. This comes at the expensive of a higher memory requirement
for the same parameter count and decreased speed.
Adds Lily: Low-Rank Interconnected Adaptation across Layers
Paper: https://arxiv.org/abs/2407.09946
Lily is on the surface similar to LoRA but has a sophisticated parameter
sharing scheme. The A parameters are shared blockwise (e.g. 4
consecutive q_proj layers share the same A). There is a pool of B
parameters that is shared globally, the actual B's are chosen in a
data-dependent way through a router. This allows Lily to use higher
ranks than LoRA while maintaining a low trainable parameter count.
Implements PSOFT: Principal Subspace Orthogonal Fine-Tuning.
Paper:
Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
https://arxiv.org/abs/2505.11235
Orthogonal fine-tuning techniques like OFT and BOFT are good at
preserving the structure and thus capabilities of the underlying base
model. PSOFT improves efficiency of this technique by constraining the
adaptation to low-rank principal subspace.
Bone was deprecated in favor of MiSS. Removal is scheduled for PEFT
v0.19.0. Old checkpoints can be converted using
scripts/convert-bone-to-miss.py.
Unrelated change: I found the skip for GPT2 for the unloading decoder
test to be unnecessary (it was probably redundant with the skip for
Conv1D layers), so that function was removed completely.
PVeRA is a probabilistic variation of VeRA, which learns a distribution in the latent adaptation space. It improves the performance of the VeRA (71.4% vs 69.9% accuracy on the VTAB-1k benchmark), and allows to use the learned latent space for uncertainty quantification (e.g. Monte Carlo confidence interval estimation).
As recommended in the issue, we based our implementation on the implementation of the VeRA adapter, as both adapters are very close.
MetaMathQA results are in line with the results above (~38% PVeRA vs. ~36% VeRA).
Implementation for https://github.com/huggingface/peft/issues/2907.
After some experimentation I think this is an implementation of the
forgetting through intruder dimensions mitigation presented by the paper
[LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://huggingface.co/papers/2410.21228).
The implementation takes a model with a loaded LoRA adapter and applies the
mitigation to form a new adapter on the same model. This makes it possible
to compare the results directly.
You can find a script to evaluate the changes here: https://gist.github.com/githubnemo/5932e99125d498c4f353017916ffc3ea
Support for `torch.compile` was considered during review but first experiments
didn't yield clear results which is why support was postponed.
## Alternative approaches
I can think of two possible modes how the conversion could be done differently.
The first one is already proposed in the original issue, namely applying the
mitigation on the merged weights. In this case we will lose the ability to
unmerge and modify the adapter directly which loses all the flexibility of
an adapter.
The second way is to modify A/B directly. Since we have the intruder dimensions
and compute the SVD of (W+dW) to build the mitigation vectors it is conceivable
that it also possible to apply the mitigation directly to the adapter's A/B
matrices. Initial testing wasn't successful so I chose the easier (but
computationally more intensive) route of first merging the weights, applying
the mitigation and then re-computing the delta weight (and subsequently the
A/B matrices using SVD of the adapter's rank).
---------
Co-authored-by: nemo <git@ningu.net>
* Bugfix turned into restructuring
Initially I wanted to fix a docs builder bug where the `loraga` autodoc path
was broken since the contents of that module were partially removed.
Then I noticed that LoRA-GA (an initialization method) had its own
`package_resource/lora_ga.md` documentation which is unusual for this kind of
method since as of now all init methods are documented in the LoRA developer guide.
Therefore, I decided to fix the bugs by a) removing the erronous paths and
b) move the package resource page of LoRA-GA to the LoRA developer guide.
Once I arrived there I noticed that aLoRA and DoRA are categorized as initialization
methods which is not true. Also QLoRA and parameter targeting were also in the
same section. These now reside in the new "Training" section. For inference
methods such as aLoRA and Arrow there is now an "Inference" section.
* Small fixes
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Visualize result via table
---------
Co-authored-by: nemo <git@ningu.net>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Adds "LoRA-GA: Low-Rank Adaptation with Gradient Approximation"
Paper link: https://arxiv.org/abs/2407.05000
Initialize the LoRA weights to approximate full rank gradient updates,
which promises better convergence.
Resolves#2651
This PR adds caching of the LoRA weight and the weight norm from DoRA
for faster inference. Since, during inference, the weights don't change,
there is no need to recalculate those weights for a DoRA module each
time.
During training, recalculation is needed, thus there is no caching when
the module has training=True.
The cache does not prevent each and every possible duplicate
calculation. For instance, the weight norm is calculated during module
initialization and then again during the first forward pass when
performing inference. Only starting from the second forward pass on will
the weight norms be cached.
The PR includes a script to measure the effect of caching. On my
machine, I get:
avg time LoRA: 0.0717 sec
avg time DoRA no caching: 0.1718 sec
avg time DoRA with caching: 0.0840 sec
memory LoRA: 15612.00 MB
memory DoRA no caching: 16212.00 MB
memory DoRA with caching: 22118.00 MB
DoRA time overhead no caching: 139.52%
DoRA time overhead with caching: 17.08%
DoRA memory overhead no caching: 3.84%
DoRA memory overhead with caching: 41.67%
Thus, caching can significantly reduce inference time but at a
noticeable cost in memory.
See discussiopn #2974.
Prefix tuning is implemented by inserting prefix embeddings into the KV
cache (past_key_values). However, in encoder-decoder models (seq2seq)
from transformers, the encoder does not make use of the KV cache, given
that it's not causal. Therefore, injecting the prefixes does not work
for the encoder, which does not correspond to the paper description
paper (https://hf.co/papers/2101.00190). This is now documented.
Prefix tuning can still be applied to encoder-decoder models and can
still learn something useful, but it's not working the way the paper
describes it.
Note that we discussed internally if encoder-decoder architectures could
be updated to allow injection via past_key_values but it would be a
non-trivial change. Given that prefix tuning of encoder-decoder models
is rather niche, we decided not to proceed with this.
This adds the possibility to convert a non-LoRA adapter into a LoRA
adapter. Not all LoRA adapters will support this, but many will.
Conversion is not precise, there will be a loss of performance. The
higher the rank, the lower the loss, but also the less efficient the
adapter. Also, for now, this only supports linear layers. Still, this
has some advantages:
- In PEFT, LoRA supports more features than most other methods, e.g.
mixed adapter batches. Thus the converted adapter can be used with those
features.
- Some downstream packages support LoRA adapters, but not other PEFT
methods, e.g. Diffusers. The conversion allows to use a non-LoRA adapter
with those packages.
Users can pass a fixed rank for the LoRA adapter or a float that will
use a dynamic rank based on the threshold of the contribution of the
singular values.
Unrelated changes
I noticed that the VB-LoRA layer had no __repr__, so it was added.
The return type annotation of set_peft_model_state_dict was incorrect.
Implements GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient
Fine-Tuning (https://arxiv.org/abs/2505.20355).
This PEFT method subdivides the base weight into smaller blocks and
applies LoRA to those. This more granular adaptation promises to
increase expressitivity and improve performance, especially at higher
ranks (64+), closing the gap to full fine-tuning.
---------
Co-authored-by: HaohanTsao <andy94729@gmail.com>
This adds a new parameter-efficient fine-tuning method called **Orthogonal Subspace Fine-Tuning (OSF)** to the PEFT library. OSF enables continual learning in LLMs by freezing the high-rank subspace of weight matrices and fine-tuning only the low-rank directions. This approach constrains updates to be orthogonal to previously important directions, thereby mitigating catastrophic forgetting without increasing parameter count.
Tracked in [PEFT Issue #2648](https://github.com/huggingface/peft/issues/2648)
**Notes**
* The current implementation does not include layerwise importance-based rank estimation (e.g., cosine similarity of inputs and activations), but can be added in future iterations
* Unmerging is not supported, as the original weights are decomposed and modified in-place
* Compared to LoRA, OSF performs a constrained update over the original weight matrix without introducing new trainable parameters, maintaining exact model architecture post-training
**Background**
This implementation is based on the method described in our paper:
Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
Implements DeLoRA: "Decoupling Angles and Strength in Low-rank
Adaptation" (https://huggingface.co/papers/2503.18225).
Similar to DoRA, DeLoRA decouples the angular learning from the
adaptation strength, but it also allows to limit the norm of the change.
This way, DeLoRA promises to reduce the risk of catastrophic forgetting
and to be more robust to hyper-parameter settings such as the learning
rate.
This PR adds the set_requires_grad method to PEFT models (both PeftModel
and BaseTuner). As the name suggests, this is a method to set the
requires_grad attribute of the specified PEFT adapters.
For more general context, this is mostly relevant when dealing with
multiple adapters. As is, users can already set the active adapter(s)
with set_adapter, which automatically adjust the requires_grad attribute
too, so that only the active adapters will have grads enabled. However,
there can be situations where activity status and requires grad may
differ. Right now, users would need to manually set requires_grad to
deal with that, which is error prone (e.g. forgetting modules_to_save).
This PR closes this gap in the API.
As this functionality is quite general purpose, I added a
set_requires_grad function to functional.py for easier integration.
Note: The set_requires_grad method will raise an error when called with
prompt learning methods like prompt tuning. This is because these
methods don't have a universal base class (BaseTuner and BaseTunerLayer)
that would allow to add this API. Moreover, they only support a single
adapter at a time, hence there is not much need to have this method in
the first place.
A side effect of not supporting prompt learning is that on the
PeftModel, we are free to allow set_requires_grad to accept more than
one adapter, which would normally be difficult, because prompt learning
only allows one adapter.
Implements the paper "Exploring Sparsity for Parameter Efficient Fine
Tuning Using Wavelets" (https://arxiv.org/abs/2505.12532).
WaveFT enables fine-grained control over the number of trainable
parameters by directly learning a sparse set of coefficients in the
wavelet domain of residual matrices. Experiments show that it works well
in the text-to-image generation space.
Deduplicate a lot of redundant code from PEFT method's model.py:
merge_and_unload
unload
delete_adapter
set_adapter
enable_adapter_layers
disable_adapter_layers
_replace_module
_unload_and_optionally_merge
_mark_only_adapters_as_trainable
_check_new_adapter_config
_check_target_module_exists
_prepare_adapter_config
__getattr__
get_peft_config_as_dict (fully deleted)
Related changes:
A new module, functional.py, is introduced, which contains functions
(just reimported from elsewhere) that can be useful for libraries that
want to integrate PEFT. I would suggest that we should treat them as
public API and thus guarantee backwards compatibility.
I also deduplicated almost identical
TRANSFORMERS_MODULES_TO_XXX_TARGET_MODULES_MAPPING constants by copying
them from LoRA and only overriding a few values that differ. Moreover,
some PEFT methods didn't have their own
TRANSFORMERS_MODULES_TO_XXX_TARGET_MODULES_MAPPING but used the one from
LoRA instead. They now each have their own constant, which is a copy
from the one from LoRA.
This PR adds support for Arrow, a modular routing mechanism for LoRA experts introduced here, as well as the refinement method GenKnowSub, proposed in our ACL 2025 Main Conference paper. GenKnowSub enhances Arrow by subtracting a general-domain LoRA from task-specific ones prior to routing, leading to improved generalisation and modularity.
Add MiSS, an evolution of Bone, from https://arxiv.org/abs/2409.15371.
MiSS will replace Bone, which is now deprecated. A script to convert Bone
checkpoints to MiSS checkpoints is included.
- Recommends trainable tokens as first measure
- Clarifies a few things about saving embeddings
- Adds full-finetuning as an option of last resort
---------
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
Add new PEFT method C³A (Circular Convolution Adaptation).
From "Parameter-Efficient Fine-Tuning via Circular Convolution":
https://arxiv.org/abs/2407.19342
Dresden