A space that looks like a web shop and allows to quickly check the
capabilities of different PEFT methods, search, and filter
results. Gives code snippets to try to use the models.
This still needs proper review, especially when it comes to the
heuristics used to determine the capabilities.
This adds the possibility to convert a non-LoRA adapter into a LoRA
adapter. Not all LoRA adapters will support this, but many will.
Conversion is not precise, there will be a loss of performance. The
higher the rank, the lower the loss, but also the less efficient the
adapter. Also, for now, this only supports linear layers. Still, this
has some advantages:
- In PEFT, LoRA supports more features than most other methods, e.g.
mixed adapter batches. Thus the converted adapter can be used with those
features.
- Some downstream packages support LoRA adapters, but not other PEFT
methods, e.g. Diffusers. The conversion allows to use a non-LoRA adapter
with those packages.
Users can pass a fixed rank for the LoRA adapter or a float that will
use a dynamic rank based on the threshold of the contribution of the
singular values.
Unrelated changes
I noticed that the VB-LoRA layer had no __repr__, so it was added.
The return type annotation of set_peft_model_state_dict was incorrect.
Add MiSS, an evolution of Bone, from https://arxiv.org/abs/2409.15371.
MiSS will replace Bone, which is now deprecated. A script to convert Bone
checkpoints to MiSS checkpoints is included.
We use ruff for linting. The version is fixed because otherwise, we
formatting changes would creep into random PRs. Thus far, the version
was ~0.6.1 but that's already quite old by now, thus moving to ~v0.9.2.
The ruff changes themselves are all about:
1. Other line breaking logic for asserts with messages
2. More aggressive string normalizaton
Comment
Making these changes is always a bit annoying since existing PRs might
need to be updated, but there is never a really good time to do it.
This change introduces CI caching for datasets and hub artifacts across runner operating systems with the intended goal to minimize the number of failed test runs because of network faults. As an additional bonus it might make the CI a bit faster.
The following artifacts are cached: ${HF_HOME}/hub/**
Note that we're avoiding .lock files as well as *.pyc files. We're not simply caching $HF_HOME since there is also the datasets and modules where the former was acting up when testing (no details, just dropped, we may explore this later but we're not using that many datasets) and the latter is just code which is probably not a good idea to cache anyway.
There is a post process for the cache action which uploads new data to the cache - only one runner can access the cache for uploading. This is done because github actions is locking cache creation, so if there's a concurrent cache creation, both may fail. This runner is currently set to ubuntu in the python 3.10 run.
If this modification turns out to be ineffective we can move to forbidding access to the hub in general (HF_HUB_OFFLINE=1) and updating the cache once per day but let's first try out if this is already enough to decrease the fail rate.
Should fix the issue of not receiving slack notifications because the
message is too long, see:
https://github.com/huggingface/peft/actions/runs/7148379741/job/19469273483
Currently, we get:
> Error: ver responded with: {'ok': False, 'error': 'invalid_blocks', 'errors': ['failed to match all allowed schemas [json-pointer:/blocks/1/text]', 'must be less than 3001 characters [json-pointer:/blocks/1/text/text]'], 'response_metadata': {'messages': ['[ERROR] failed to match all allowed schemas [json-pointer:/blocks/1/text]', '[ERROR] must be less than 3001 characters [json-pointer:/blocks/1/text/text]']}}
Fixing the error should also lead to a shorter message, but we should
ensure that even if the message is too long, we still get it.
Previously, we imported from bitsandbytes eagerly if the package was
installed. This caused two major issues:
- Slow loading time of PEFT (~4 sec)
- Errors with multiprocessing because bnb initializes CUDA
This commit fixes both issues by importing bitsandbytes lazily. PEFT
import time is now reduced to ~2sec.
Notes
Implementation-wise, I use a combination of local imports and
module-level __getattr__. The latter was introduced in Python 3.7 and
should therefore be safe to use.
Benjamin Bossan