3f8b44a76d
Merge pull request #1595 from hanhainebula/master
...
Fix Transformers 5 tokenizer compatibility
2026-08-24 10:48:44 +08:00
71fa99750a
Fix Transformers 5 tokenizer compatibility
2026-08-24 10:44:25 +08:00
7b7be6748d
Merge pull request #1594 from hanhainebula/master
...
fix(reranker): normalize model mapping names
2026-08-23 20:36:55 +08:00
0752381a23
fix(reranker): normalize model mapping names
2026-08-23 20:35:16 +08:00
807fc3fcc5
Merge pull request #1593 from hanhainebula/master
...
fix(reranker): support transformers v5 tokenizer preparation
2026-08-23 20:26:09 +08:00
0e924f5205
fix(reranker): support transformers v5 tokenizer preparation
2026-08-23 20:17:50 +08:00
a26f9ef85f
Merge pull request #1592 from hanhainebula/master
...
Fix Transformers 5 Trainer compatibility
2026-08-23 19:30:08 +08:00
5894f91732
Fix Transformers 5 Trainer compatibility
2026-08-23 18:58:54 +08:00
b5a492d86c
Fix Gemma reranker imports with Transformers v5
2026-05-19 00:05:59 +08:00
d82edd1ed1
fix: add DEFAULT_POOLING_METHOD to PseudoMoELLMEmbedder
2026-04-02 22:06:41 +08:00
ab7c252d62
feat(embedder): add decoder-only pseudo_moe inference with domain routing support
...
- pass domain_for_pseudo_moe through evaluation runner to embedder initialization
- add new decoder-only embedder implementation for pseudo MoE models
2026-04-01 22:22:01 +08:00
ac3e2390e0
feat: add use_mrl interface of finetune for embedder decode_only.icl and encoder_only.base
2026-03-31 21:15:00 +08:00
29d380b9e1
feat: add truncate_dim for evaluation and inference for embedder
2026-03-31 17:10:51 +08:00
b09c37751d
feat: add the 'use_mrl' interface to the finetune of embedder
2026-03-31 15:35:31 +08:00
2ff964ee32
fix(eval): pass use_bf16 through AbsEvalRunner.get_model and COIR evaluator model initialization
2026-03-26 20:59:03 +08:00
2955a5a6e5
fix(m3): make bf16 inference/train loading safe and unify numpy conversion
...
- replace m3 embedder .cpu().numpy() paths with base _convert_to_numpy(...) for bf16-safe conversion
- add torch_dtype plumbing in m3 runner model loading (AutoModel/colbert_linear/sparse_linear) to keep dtype behavior consistent
2026-03-26 19:14:12 +08:00
d65ffb17b7
fix(embedder): add _convert_to_numpy in base class and guard bf16->numpy on non-cpu
2026-03-26 15:19:05 +08:00
19edba7b08
feat: fix the interface of attn_implementation in embedder.decode_only.*.load_model and reranker.decode_only.*.load_model
2026-03-26 14:17:14 +08:00
6679caaf65
fix: code format for inference.embedder.decoder_only.icl
2026-03-26 14:00:41 +08:00
5d6fa9a69e
fix: add bf16 interface to EncoderOnlyEmbedderM3Runner.get_model
2026-03-26 13:37:05 +08:00
90957832ac
feat(embedder): add use_bf16 support and unify inference dtype behavior
...
add use_bf16 to auto embedder and all embedder constructors
2026-03-26 12:17:14 +08:00
94993fd299
added init.py to fix import errors
2026-03-09 18:12:33 +00:00
3893bf2e4e
feat: add compatibility with transformers V5
2026-03-05 14:48:52 +00:00
d166f499de
add metric for evaluation: evaluate_recall_cap
2025-12-15 16:50:02 +08:00
a1befaa641
support evaluation: BRIGHT
2025-10-10 19:17:30 +08:00
53240c5283
support evaluation: BRIGHT
2025-10-10 16:50:42 +08:00
24bc8d15a6
fix bug: safe dist.get_rank()
2025-09-23 23:02:26 +08:00
ffea23c79d
Update model_mapping for embedder inference
...
- bge-code-v1, bge-reasoner-embed-qwen3-8b-0923
- Qwen3-Embedding-*
2025-09-23 22:51:11 +08:00
17d8e8a566
fix abc/evaluation/runner.py: evaluate_metrics
...
- when eval_results_path does not exist, continue
2025-09-22 20:35:28 +08:00
f23392228a
add use_fast_tokenizer param for finetuning
...
- default: True
2025-09-17 16:18:53 +08:00
3bc1962480
fix bug: only modify vocab_size when necessary
...
- when saving ckpt for decoder only embedder, check `emb.pth`
2025-07-15 10:33:52 +08:00
90173852db
disable warning info: train_group_size is None
2025-07-13 00:47:57 +08:00
15876a9391
add feature: support customized train_group_size
2025-07-12 17:55:03 +08:00
bf410d51e9
update environment
2025-05-28 14:55:54 +08:00
02375dbda2
fix a bug in evaluation/mteb/runner.py
2025-04-15 14:37:58 +08:00
10183c4890
Merge pull request #1421 from tuobay/patch-1
...
Update runner.py
2025-04-15 10:14:21 +08:00
6eefbac0e0
fix m3 sparse embedding bug
...
- using optimization suggestion from issue #1364 only when not training
2025-04-14 18:59:47 +08:00
7662fb343d
refactor code: update typing for parameters
...
- transformers:
- AutoModel -> PreTrainedModel
- AutoTokenizer -> PreTrainedTokenizer
- FlagEmbedding:
- FlagAutoModel -> AbsEmbedder
- FlagAutoReranker -> AbsReranker
2025-04-14 17:48:43 +08:00
e6b9488f57
fix a bug in m3 modeling.py: #1410
2025-04-10 20:51:41 +08:00
dd8ccc9124
Merge pull request #1401 from pfldy2850/patch-1
...
Change fixed value `256` in `trange` to `batch_size`
2025-04-10 17:00:58 +08:00
94068b27b3
Update base.py
2025-04-10 16:59:53 +08:00
7a0a03a85e
Update lightweight.py
2025-04-10 16:59:21 +08:00
cc978ae106
Update layerwise.py
2025-04-10 16:59:01 +08:00
4e24ad45e4
Update base.py
2025-04-10 16:58:37 +08:00
df8b48c493
Update m3.py
2025-04-10 16:57:49 +08:00
dda2002d12
Update icl.py
2025-04-10 16:56:56 +08:00
c43a3c6759
Update base.py
2025-04-10 16:55:15 +08:00
0f8b0d80fc
Update base.py
2025-04-10 16:43:23 +08:00
aa0dceee74
Merge pull request #1378 from listeng/add-missing-param
...
add missing trust_remote_code[=true] when loading "openbmb/MiniCPM-Reranker-Light"
2025-04-10 16:29:09 +08:00
4e82b52649
optimze m3: sparse_embedding, refer to #1364
2025-04-10 16:13:26 +08:00
Jianlyu Chen