Expand contributing installation guide with fork and setup steps
Quote editable install and link contributing guide from install.md
Fix relative path of contributing link in install.md
Resolves#3340
Context
So far, we did not allow adding multiple LoRA adapters with
target_parameters on the same layer. This was a known limitation. I
have already attempted to solve this once (see #2710) but didn't have
time to come up with a nice solution. As it was unclear if there was
any real world need to support this, there was no further work on it
since then. Now we know that there are practical application that may
need it, so I resumed the work.
What doesn't work
The previous solution attempted to solve the issue by nesting the
lora.ParamWrappers. So for adapters 'default' and 'other', we would
end up having something like:
param_wrapper_default(param_wrapper_other(base_layer))
This was problematic. Not only could this result in very deep nesting,
which is inefficient. What's worse is that state_dict key for 'other'
would contain 'base_layer.' as an infix. Therefore, if we wanted to
load the 'other' adapter _without_ first loading the 'default'
adapter, we would get a key mismatch.
We could also not simply strip out 'base_layer.' infix because we use
nesting to deal with multiple nn.Parameters on the same module, so to
account for that, we need to keep the infix.
Solution
The solution is pretty straightfoward: We use the existing mechanism
to store the parameters for the other adapter in the
nn.ModuleDict. For this, we detect if the layer is already a
ParamWrapper when adding the second adapter and update that layer
instead of nesting it.
Caveat
This simple approach can, however, not work with multiple adapters
that target a different set of parameters. This is because the
information which parameter is targeted is not stored in the
state_dict itself. Therefore, if we had different adapters targeting
different parameters, we would not be able to tell which parameter is
meant to be targeted.
Adds DEFT: "DEFT: Decompositional Efficient Fine-Tuning for
Text-to-Image Models"
https://arxiv.org/abs/2509.22793
DEFT splits a weight update into two learned low-rank parts: a
projection that removes a sub-space of the frozen weight, and a low-rank
update that injects new content in its place. This allows DEFT to adapt
new data or concepts, e.g. personalizing a text-to-image model from a
few images, while retaining the base model's
instruction-following/editability with minimal forgetting. It's less
suitable if you don't need to preserve the base model's other
capabilities or for layers beyond Linear/Conv1D.
PaRa ("Personalizing Text-to-Image Diffusion via Parameter Rank
Reduction") is also supported as a special case by passing para=True.
https://arxiv.org/abs/2406.05641
In the doc navigation, under "LoRA > Initialization", there is a
"Usage Tips" section but clicking it doesn't work. The reason is that
it's a subsection of LoRA-GA, but LoRA-GA is not open by default as
the initialization options are formatted as different tabs.
My proposal is to remove the section title so that it no longer
appears in the navigation.
## Motivation
This PR adds **UniLoRA**, a LoRA-style parameter-efficient fine-tuning method
that introduces a unified parameterization for low-rank adaptations, enabling
further reductions in the number of trainable parameters while preserving
the standard PEFT workflow.
## What's included
- UniLoRA tuner implementation
- Configuration class and registry integration
- Save/load support
- Unit tests
## Blurb for publication
UniLoRA shares a compact trainable vector bank across low-rank adapter weights. It keeps the familiar PEFT training flow while using deterministic projections into shared `theta_d` values to reduce the number of trained adapter parameters.
Adds GLoRA: "One-for-All: Generalized LoRA for Parameter-Efficient
Fine-tuning"
https://huggingface.co/papers/2306.07967
GLoRA (Generalized LoRA) is a flexible PEFT method that extends LoRA
with configurable weight, activation, and bias adaptation, delivering
richer fine-tuning with no extra inference cost. Use it when you need
per-layer flexibility or stronger adaptation than vanilla LoRA. Skip it
for non-Linear layers (e.g. Conv/Embedding) or when standard LoRA is
already sufficient and simplicity matters.
Add: MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning
(https://arxiv.org/abs/2604.01694)
MiCA (Minor Component Adaptation) is a new initialization option for
LoRA. It initializes the LoRA B weight to the minor components of the
base weight and then freezes it, only learning A. This promises to
improve knowledge acquisition while reducing catastrophic forgetting and
reducing the number of parameters that need to be trained.
The question recently came up if hotswapping works with
target_parameters. Therefore, I added a test to check it. It turns out
that it works indeed.
The usefulness is, however, somewhat reduced because targeting
parameters while using torch.compile (compiled models are a main use
case for hotswapping) leads to re-compilation and/or graph
breaks. This is a fundamental limitation of how targeting
nn.Parameters is implemented, using nn.utils.parametrize to
dynamically update the targeted nn.Parameter. We can't update it
statically, since that would break all kinds of things (e.g. accessing
the parameter with model.foo.bar would return the parameter *after*
applying the LoRA delta weight). Therefore, we must undo the
parametrization after the forward step, and this breaks compilation.
This PR additionally documents the fundamental problem with
torch.compile and target_parameters. It also removes an unused
argument in a test and an incorrect comment.
Implements: FRoD: Full-Rank Efficient Fine-Tuning with Rotational
Degrees for Fast Convergence
https://arxiv.org/abs/2512.23485
FRoD is a full-rank, replacement-style PEFT method from FRoD: Full-Rank
Efficient Fine-Tuning with Rotational Degrees for Fast
Convergence. Instead of adding low-rank deltas, it reconstructs selected
weights with shared rotational subspaces and sparse trainable
coefficients. It is especially useful when fast convergence and a higher
full-rank capacity ceiling are important, and its large sparse
rotational subspace may also be promising for model merging. The main
tradeoffs are the costly joint-decomposition initialization and slightly
slower forward/backward passes than LoRA due to the sparse structured
factors, so it may be less attractive for a one-off single-task
fine-tune.
This is a minor change on top of #3300: the API reference should
be sorted in the same way the methods are sorted (alphabetically)
and since it is uncurated it should be collapsed by default.
The section "adapters" is now called "tuners" to more adequately
model the API.
Co-authored-by: nemo <git@ningu.net>
The current state of the PEFT docs is not one of structure and I was constantly annoyed that whenever I wanted to change something there were several places that needed touching and they all felt disconnected. So this is my attempt at structuring the docs. Some of these ideas are quite old (discussed in 01/2025) but are still valid.
I've removed most of the code guides without replacement. That's not ideal, I think we should have code examples but I'm think they should be method-focused. Maybe one general example of a training workflow is sufficient because most methods follow the same scheme.
All details from the method guides (prompting, lora, oft/boft, etc.) are now integrated into the respective method pages instead. I would have hesitated to do this if these guides would have integrated information about the adapters but they didn't. I think it makes a lot more sense to have one place for each method to gather examples/tips/recommendations and that is now the `package_refernce/<method>` page. This page now also hosts a small space that shows the MetaMathQA (and potentially other) benchmark results highlighted for that method.
I've moved the LoRA initializations to `package_reference/lora#Initialization` and converted the init methods to `<hfoption>`-tags. This collapses them to a list but may reduce searchability through the document - at least firefox is not able to search 'through' the option tabs. This also doesn't make them appear in the ToC and people specifically searching for, say, PiSSA won't find it directly. I think that's OK though, since the search is able to locate it.
The quicktour is a bit more detailed about what happens under the hood (quick doesn't have to mean simplistic) and includes some new visualizations. I hope that we can integrate more visualizations in the future where it makes sense.
* Remove PEFT method space + front page buttons
The space was not that useful anymore since most methods are compatible
with most models.
The front page buttons are, at least temporarily, with the exception
of the quicktour and method overview buttons. I like the visuals
but there should only be elements that are useful.
---------
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Problem description
Right now, if a new PEFT method wants to add support for quantized
layers, it requires a significant amount of work. Notably, the method
needs to implement dedicated layer classes for each quantization method
(e.g. one class for bnb 4bit, one for bnb 8bit, one for AWQ, ...). These
classes typically are >90% boilerplate and the actual difference between
implementations of these classes is minimal.
The result of that is that, at the moment, most PEFT methods don't
support any, or only very few, quantization methods, even though the
amount of actual logic required to support these methods is relatively
small.
Suggested solution
This PR is a suggestion of how to solve the issue. With a few extra
lines, we should be able to support all quantization methods in all PEFT
methods. The general approach is to add an attribute to each PEFT layer,
self.quantization_backend, which supports these methods:
- get_base_weight - set_base_weight
When the PEFT layers use these methods to access and write to the base
layer weight, and if the weight is quantized, the new classes will deal
with that correctly. This means that we no longer need a dedicated layer
class to deal with quantized layers, the normal layer class will do.
E.g. for MiSS, the normal miss.Linear class can deal with bnb layers,
there is no need to add a miss/bnb.py module with dedicated layers.
A few rewrites in the existing PEFT methods are required to support this
new quantization backend class, but the amount of total code needed for
that is considerably smaller than adding new classes for each
quantization method.
Furthermore, these quantization backend classes are agnostic with regard
to the PEFT method. Therefore, with M PEFT methods and N quantization
methods, we no longer need MxN implementations to support quantization
but only M+N.
Migration
For LoRA, we have already implemented the layer classes for each
supported quantization method. For the sake of consistency, it could
still make sense to migrate LoRA to the new approach if it's accepted.
This needs to be accompanied by detailed regression testing to ensure
that everything keeps working. I would only suggest to deprecate and
remove abandoned quantization methods (perhaps for a v1.0 release).
Scope
Updating all PEFT methods is too much for a single PR. This PR focuses
on only three PEFT methods for now:
- MiSS: A pretty normal PEFT method, representative of many other PEFT
methods. - BOFT: Also pretty normal, but requires slight rewrite of the
forward step. Similar changes may be required for other methods too. -
VeRA: Already supports bnb but with this PR, specific BNB layers are no
longer needed.
Implements MonteCLoRA: "Robust and Efficient Fine-tuning of LLMs with
Bayesian Reparameterization of Low-Rank Adaptation"
https://huggingface.co/papers/2411.04358
This LoRA variant augments the LoRA A matrix by including Monte Carlo
estimations of the LoRA parameters. This promises to make the model
more robust to hyper-parameter choice and to lead to better
regularization, at the cost of a small increase in the count of
learnable parameters. There is no extra cost at inference time compared
to normal LoRA.
Implements "VeLoRA: Memory Efficient Training using Rank-1 Sub-Token
Projections"
(https://huggingface.co/papers/2405.17991)
VeLoRA is a LoRA variant that reduces memory required for training by
compressing the activations saved for the LoRA in the forward pass and
then reconstructing them in the backwards pass. Compared to gradient
checkpointing, VeLoRA is faster but requires more memory. The LoRA
adapter itself is unaffected and can be used like any normal LoRA
adapter.
Implements KappaTune based on:
"The Condition Number as a Scale-Invariant Proxy for Information Encoding in
Neural Units"
https://arxiv.org/abs/2506.16289
This is a helper function that can be run on the base model to identify the
layers that are most suited for fine-tuning. It works for Linear and MoE
layers. By targeting these layers with LoRA or other PEFT methods, the model
should train well without forgetting useful information from pre-training.
Adds "HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large
Language Models" (https://openreview.net/pdf?id=TwJrTz9cRS)
This PEFT method is similar to LoRA but instead of updates to the base
weights being additive, they are multiplicative here (Hadamard product).
This promises to resolve some limitations of the low rank updates
provided by LoRA, especially for learning tasks requiring high
expressivity. Convolutional and embedding layers, as well as
bitsandbytes quantization, are supported.
Adds: "BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data
Regimes"
Paper: https://arxiv.org/abs/2509.15974
BEFT only learns the bias terms of the targeted layers, making it
extremely parameter efficient. Results show that BEFT works especially
well when targeting the V projection in low data regimes. When there is
a lot of training data and higher learning capacity is required, BEFT is
not the best fit.
Now that the PRs in relation to #2864 have been merged, we should also
properly document weight tying behavior in PEFT. I added a section to
the docs based on said issue.
At inference, PEFT can result in a substantial runtime overhead when
targeting MoE parameters. This is now documented, as well as how to
avoid the overhead.
Especially the section on adding new PEFT methods was incomplete and
should be more actionable now.
With the arrival of coding agents, there is even the small chance that
someone might read it.
While optimizing the hyper-parameters for prefix tuning in the MetaMathQA benchmark
most of the results turned out to have ~0% task accuracy (except for the baseline
configuration). Initializing the prefixes to be a no-op in the beginning, similar to
LoRA's default initialization, turned out to resolve this:
* Baseline: ~20% task accuracy
* Zero-init: ~35% task accuracy
The benchmark setup did not allow for passing an KV cache initialization string
which this change also corrects. Initializing the KV cache with a simple string
like "Question:" reaches a similar task accuracy (~36%) while having a lot less
forgetting than zero init.
A longer, task specific sequence (50 tokens) reaches ~43% task accuracy and
~0.5 forgetting.
Implements "AdaMSS: Adaptive Multi-Subspace Approach for
Parameter-Efficient Fine-Tuning"
https://openreview.net/forum?id=8ZdWmpYxT0
AdaMSS segments the base weights of the model into smaller subspaces
that are targeted for fine-tuning. Moreover, it's possible to
dynamically assign a lower parameter budget to less important subspaces
during training, similar to what AdaLoRA does. This promises to provide
higher expressiveness and better generalization than similar PEFT
methods.
Adds TinyLoRA, a new PEFT method based on "TinyLoRA: Learning to Reason in 13 Parameters". TinyLoRA achieves extreme parameter efficiency by replacing LoRA's trainable low-rank matrices with a tiny trainable vector projected through fixed random bases.
The key idea: given a frozen SVD decomposition `W ≈ B @ A` (where `B = U @ sqrt(S)` and `A = sqrt(S) @ V^T`), the weight update is `delta_W = B @ R @ A` where `R` is an `r x r` trainable matrix (following LoRA-XS). TinyLoRA takes this further by parameterizing `R` as a linear combination of fixed random projection matrices:
R = sum_i(v[i] * P[i])
where `v` is the only trainable parameter (as small as 13 values) and `P_i` are fixed random matrices seeded deterministically.
## Features
- Extreme efficiency: trainable parameter count is `u` per target module (or even less with weight tying), compared to `r * (in + out)` for LoRA
- Weight tying: configurable sharing of `v` vectors across layers via `weight_tying` (0.0 = no sharing, 1.0 = all layers share one `v`)
- SVD initialization: frozen `A` and `B` matrices computed from truncated SVD of pretrained weights, with singular values distributed equally via `sqrt(S)`
- Full layer support: `nn.Linear`, `Conv1D`, and `nn.Embedding`
- Merge/unmerge: full support including safe merge with NaN checking
- LoRA conversion: `supports_lora_conversion()` -> True — delta weights can be converted to standard LoRA format via `get_delta_weight`
- Deterministic projections: `P` matrices are seeded per-layer for reproducibility; optionally saved in checkpoints (`save_projection=True`)
## Config
```
from peft import TinyLoraConfig, get_peft_model
config = TinyLoraConfig(
r=2, # SVD rank (frozen)
u=64, # trainable vector dimension
weight_tying=0.0, # 0.0=no sharing, 1.0=full sharing
target_modules="all-linear",
)
model = get_peft_model(base_model, config)
```
## Architecture
- TinyLoraLayer (base): SVD decomposition, projection init, `get_delta_weight`, `supports_lora_conversion`
- Linear / Embedding: forward pass, merge/unmerge
- TinyLoraModel: weight tying groups, shared v parameter management via nested ModuleDict/ParameterDict
- update_layer follows LoRA's config-object pattern: (adapter_name, tinylora_v, v_key, r, config, **kwargs)
---------
Co-authored-by: githubnemo <githubnemo@users.noreply.github.com>
Torchao made some API changes, which have to be reflected in the tests.
Moreover, for this to pass, we also need transformers to make the
corresponding adjustments:
https://github.com/huggingface/transformers/pull/44604
While working on this, I migrated the tests from unittest to pytest
style.
DeloRA is missing some config validation that other tuners already have (exclude_modules list-to-set conversion, r validation, Literal type for bias). This PR fixes that.
Also fixes some typos I noticed while going through the code.
The financial_phrasebank dataset fails to load with recent versions of
the datasets library due to deprecation of loading scripts. Replace it
with zeroshot/twitter-financial-news-sentiment which is a compatible
financial sentiment dataset available in the new parquet format.
Add PEANut: Parameter-Efficient Adaptation with Weight-aware Neural
Tweakers
Paper: https://arxiv.org/abs/2410.01870
PEANuT adds a small neural net (weight-aware neural tweakers). Compared
to LoRA, this increases expressivity for the same trainable parameter
count or allows to greatly lower the parameter count without sacrificing
expressivity. This comes at the expensive of a higher memory requirement
for the same parameter count and decreased speed.
There's no official rule how we regard typo PRs. To give contributors
a clear vision of how we want typo PRs to be handled I suggest this
update to the guidelines.
Adds Lily: Low-Rank Interconnected Adaptation across Layers
Paper: https://arxiv.org/abs/2407.09946
Lily is on the surface similar to LoRA but has a sophisticated parameter
sharing scheme. The A parameters are shared blockwise (e.g. 4
consecutive q_proj layers share the same A). There is a pool of B
parameters that is shared globally, the actual B's are chosen in a
data-dependent way through a router. This allows Lily to use higher
ranks than LoRA while maintaining a low trainable parameter count.
Implements PSOFT: Principal Subspace Orthogonal Fine-Tuning.
Paper:
Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
https://arxiv.org/abs/2505.11235
Orthogonal fine-tuning techniques like OFT and BOFT are good at
preserving the structure and thus capabilities of the underlying base
model. PSOFT improves efficiency of this technique by constraining the
adaptation to low-rank principal subspace.
Bone was deprecated in favor of MiSS. Removal is scheduled for PEFT
v0.19.0. Old checkpoints can be converted using
scripts/convert-bone-to-miss.py.
Unrelated change: I found the skip for GPT2 for the unloading decoder
test to be unnecessary (it was probably redundant with the skip for
Conv1D layers), so that function was removed completely.
PVeRA is a probabilistic variation of VeRA, which learns a distribution in the latent adaptation space. It improves the performance of the VeRA (71.4% vs 69.9% accuracy on the VTAB-1k benchmark), and allows to use the learned latent space for uncertainty quantification (e.g. Monte Carlo confidence interval estimation).
As recommended in the issue, we based our implementation on the implementation of the VeRA adapter, as both adapters are very close.
MetaMathQA results are in line with the results above (~38% PVeRA vs. ~36% VeRA).
To me the LoftQ application process was not immediately obvious.
I'm sure it can't hurt to add a bit of context what the two main
ways are to apply LoftQ.
In my quest to search for enlightenment I was often directed to the
quantization guide in the PEFT docs which featured a lot less details
than the developer guide for LoRA. Since LoftQ is mainly used in quantization
I think it makes sense to move it there and link there from the developer guide.
Add docs on how to set the rank of LoRA adaptors to the experts in MOEs to be the rank / total experts
---------
Co-authored-by: githubnemo <githubnemo@users.noreply.github.com>
Implementation for https://github.com/huggingface/peft/issues/2907.
After some experimentation I think this is an implementation of the
forgetting through intruder dimensions mitigation presented by the paper
[LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://huggingface.co/papers/2410.21228).
The implementation takes a model with a loaded LoRA adapter and applies the
mitigation to form a new adapter on the same model. This makes it possible
to compare the results directly.
You can find a script to evaluate the changes here: https://gist.github.com/githubnemo/5932e99125d498c4f353017916ffc3ea
Support for `torch.compile` was considered during review but first experiments
didn't yield clear results which is why support was postponed.
## Alternative approaches
I can think of two possible modes how the conversion could be done differently.
The first one is already proposed in the original issue, namely applying the
mitigation on the merged weights. In this case we will lose the ability to
unmerge and modify the adapter directly which loses all the flexibility of
an adapter.
The second way is to modify A/B directly. Since we have the intruder dimensions
and compute the SVD of (W+dW) to build the mitigation vectors it is conceivable
that it also possible to apply the mitigation directly to the adapter's A/B
matrices. Initial testing wasn't successful so I chose the easier (but
computationally more intensive) route of first merging the weights, applying
the mitigation and then re-computing the delta weight (and subsequently the
A/B matrices using SVD of the adapter's rank).
---------
Co-authored-by: nemo <git@ningu.net>
* Bugfix turned into restructuring
Initially I wanted to fix a docs builder bug where the `loraga` autodoc path
was broken since the contents of that module were partially removed.
Then I noticed that LoRA-GA (an initialization method) had its own
`package_resource/lora_ga.md` documentation which is unusual for this kind of
method since as of now all init methods are documented in the LoRA developer guide.
Therefore, I decided to fix the bugs by a) removing the erronous paths and
b) move the package resource page of LoRA-GA to the LoRA developer guide.
Once I arrived there I noticed that aLoRA and DoRA are categorized as initialization
methods which is not true. Also QLoRA and parameter targeting were also in the
same section. These now reside in the new "Training" section. For inference
methods such as aLoRA and Arrow there is now an "Inference" section.
* Small fixes
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Visualize result via table
---------
Co-authored-by: nemo <git@ningu.net>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Adds "LoRA-GA: Low-Rank Adaptation with Gradient Approximation"
Paper link: https://arxiv.org/abs/2407.05000
Initialize the LoRA weights to approximate full rank gradient updates,
which promises better convergence.
Resolves#2651
This PR adds caching of the LoRA weight and the weight norm from DoRA
for faster inference. Since, during inference, the weights don't change,
there is no need to recalculate those weights for a DoRA module each
time.
During training, recalculation is needed, thus there is no caching when
the module has training=True.
The cache does not prevent each and every possible duplicate
calculation. For instance, the weight norm is calculated during module
initialization and then again during the first forward pass when
performing inference. Only starting from the second forward pass on will
the weight norms be cached.
The PR includes a script to measure the effect of caching. On my
machine, I get:
avg time LoRA: 0.0717 sec
avg time DoRA no caching: 0.1718 sec
avg time DoRA with caching: 0.0840 sec
memory LoRA: 15612.00 MB
memory DoRA no caching: 16212.00 MB
memory DoRA with caching: 22118.00 MB
DoRA time overhead no caching: 139.52%
DoRA time overhead with caching: 17.08%
DoRA memory overhead no caching: 3.84%
DoRA memory overhead with caching: 41.67%
Thus, caching can significantly reduce inference time but at a
noticeable cost in memory.
See discussiopn #2974.
Prefix tuning is implemented by inserting prefix embeddings into the KV
cache (past_key_values). However, in encoder-decoder models (seq2seq)
from transformers, the encoder does not make use of the KV cache, given
that it's not causal. Therefore, injecting the prefixes does not work
for the encoder, which does not correspond to the paper description
paper (https://hf.co/papers/2101.00190). This is now documented.
Prefix tuning can still be applied to encoder-decoder models and can
still learn something useful, but it's not working the way the paper
describes it.
Note that we discussed internally if encoder-decoder architectures could
be updated to allow injection via past_key_values but it would be a
non-trivial change. Given that prefix tuning of encoder-decoder models
is rather niche, we decided not to proceed with this.
Dresden