71fa99750a
Fix Transformers 5 tokenizer compatibility
2026-08-24 10:44:25 +08:00
0e924f5205
fix(reranker): support transformers v5 tokenizer preparation
2026-08-23 20:17:50 +08:00
5894f91732
Fix Transformers 5 Trainer compatibility
2026-08-23 18:58:54 +08:00
ab7c252d62
feat(embedder): add decoder-only pseudo_moe inference with domain routing support
...
- pass domain_for_pseudo_moe through evaluation runner to embedder initialization
- add new decoder-only embedder implementation for pseudo MoE models
2026-04-01 22:22:01 +08:00
29d380b9e1
feat: add truncate_dim for evaluation and inference for embedder
2026-03-31 17:10:51 +08:00
b09c37751d
feat: add the 'use_mrl' interface to the finetune of embedder
2026-03-31 15:35:31 +08:00
2ff964ee32
fix(eval): pass use_bf16 through AbsEvalRunner.get_model and COIR evaluator model initialization
2026-03-26 20:59:03 +08:00
d65ffb17b7
fix(embedder): add _convert_to_numpy in base class and guard bf16->numpy on non-cpu
2026-03-26 15:19:05 +08:00
90957832ac
feat(embedder): add use_bf16 support and unify inference dtype behavior
...
add use_bf16 to auto embedder and all embedder constructors
2026-03-26 12:17:14 +08:00
d166f499de
add metric for evaluation: evaluate_recall_cap
2025-12-15 16:50:02 +08:00
24bc8d15a6
fix bug: safe dist.get_rank()
2025-09-23 23:02:26 +08:00
17d8e8a566
fix abc/evaluation/runner.py: evaluate_metrics
...
- when eval_results_path does not exist, continue
2025-09-22 20:35:28 +08:00
f23392228a
add use_fast_tokenizer param for finetuning
...
- default: True
2025-09-17 16:18:53 +08:00
90173852db
disable warning info: train_group_size is None
2025-07-13 00:47:57 +08:00
15876a9391
add feature: support customized train_group_size
2025-07-12 17:55:03 +08:00
7662fb343d
refactor code: update typing for parameters
...
- transformers:
- AutoModel -> PreTrainedModel
- AutoTokenizer -> PreTrainedTokenizer
- FlagEmbedding:
- FlagAutoModel -> AbsEmbedder
- FlagAutoReranker -> AbsReranker
2025-04-14 17:48:43 +08:00
ca91f2b5d1
fix a bug in embedder finetune:AbsEmbedderCollator
2025-04-10 16:06:31 +08:00
21311c64c6
fix bug: fix stop_self_pool function for inference
2025-03-20 18:48:58 +08:00
1bfdf2cbb0
fix "\\n" issue: replace "\\n" with "\n"
2025-02-13 22:41:54 +08:00
3b09d800aa
optimize evaluation process in evaluator.py
...
- AbsEvaluator
- BEIREvaluator
2025-02-10 16:23:35 +08:00
44e552575f
Merge pull request #1350 from qiyulei-mt/musa_support
...
support musa backend in FlagEmbedding
2025-02-07 21:30:46 +08:00
6c9dba5fe1
add check hasattr(torch, "musa")
2025-02-07 21:28:01 +08:00
c733ecb5a3
add check hasattr(torch, "musa")
2025-02-07 21:26:30 +08:00
a0e42945a2
del dp code
2025-02-07 21:20:38 +08:00
4ffa194839
support musa backend in FlagEmbedding
2025-01-23 10:35:05 +08:00
f18806c90a
Merge branch 'FlagOpen:master' into master
2025-01-22 15:20:33 +08:00
ddf9adaee0
fix bug: _concatenate_results_from_multi_process
...
move all tensor to the same device
2025-01-13 14:44:44 +08:00
b83f16abd3
fix list[BatchEncoding] typehint error
...
def __call__(self, features) -> list[BatchEncoding]:
TypeError: 'type' object is not subscriptable
cannot work in python3.8
change into List[BatchEncoding]:
2024-12-20 14:58:31 +08:00
0874413929
update dp
2024-11-29 17:25:05 +08:00
acd2bdafb8
update stop_self_pool function to avoid Exception
...
- embedder and reranker
2024-11-22 16:11:23 +08:00
5dcecd664a
rm useless __main__.py in abc/inference
2024-11-22 11:19:35 +08:00
fde4abdb48
Merge pull request #1227 from 545999961/master
...
update mteb eval
2024-11-15 12:16:36 +08:00
61ab7e0e5c
update mteb eval
2024-11-15 12:14:15 +08:00
7caa5989a7
eval abc
2024-11-14 13:38:16 +08:00
a70e1b2d07
empty cuda
2024-11-14 12:33:51 +08:00
d6815745c8
update stop pool
2024-11-14 12:26:30 +08:00
c9026b405e
fix a bug in AbsDataset.py for embedder training
2024-11-14 08:52:43 +08:00
8c3e1ebc87
fix bugs in data_loader.py for evaluation
...
- missed parameters: dataset_name
2024-11-14 01:09:19 +08:00
04a9dc378f
fixed a bug in AbsReranker.py for mps device support
2024-11-12 21:10:59 +08:00
df34cae1a8
fix a bug in AbsDataset.py for embedder finetune
2024-11-12 12:27:12 +08:00
3f071736d4
evaluation
2024-11-11 17:29:15 +08:00
d2c4a681c6
update distill
2024-11-08 15:32:54 +08:00
235a775568
ft base & m3
2024-11-07 15:25:13 +00:00
834498f2e4
abs ft reranker
2024-11-07 15:25:13 +00:00
f43b46d0a4
embedder finetune
2024-11-07 15:25:13 +00:00
9ff1fa98fa
embedder ft abc
2024-11-07 15:25:13 +00:00
eade969599
docstring fix and reranker
2024-11-04 09:44:41 +00:00
6f693a232d
base & m3
2024-11-04 04:27:51 +00:00
ac3a9ce2c6
noindex
2024-11-04 04:27:51 +00:00
f5c3891e65
docstring modification
2024-11-04 04:27:51 +00:00
hanhainebula