Abdullah Alaqeel
8c90741c63
* chore(deps): migrate @xenova/transformers to @huggingface/transformers v4 @xenova/transformers@2.x is deprecated and silently broken on Node 22+ (see #479). The project was renamed to @huggingface/transformers; same Apache-2.0 license, same code. v4 ships onnxruntime-node/web and sharp as hard deps, so they're dropped from our optionalDependencies. Pipeline / RawImage.fromBlob / tolist / text-classification output shape all unchanged. Three behavior-preserving adjustments needed: - All 4 pipeline call sites pass { dtype: "q8" }. v4's default on Node is fp32 (DEFAULT_DEVICE_DTYPE = "fp32"); v2 defaulted to quantized=true. Without explicit dtype, all 4 sites silently regress (~3.5x larger download, slower inference). dtype "q8" maps to model_quantized.onnx per v4's DEFAULT_DTYPE_SUFFIX_MAPPING; file exists in all 3 Xenova models. This was the regression that prompted the test additions below. - src/providers/embedding/local.ts: split import try/catch from pipeline() call so model-load errors (network, missing q8 variant, disk) propagate with their actual message, not masked as "Install @huggingface/transformers...". - src/providers/embedding/{local,clip}.ts: type module from typeof import("@huggingface/transformers") so PretrainedModelOptions flows through; drop hand-rolled aliases and @ts-ignore. Cast at assignment sites (pipeline return union isn't structurally assignable to our narrow FeatureExtractor / ClipPipeline shapes). Tests added where coverage was zero (would have caught the dtype regression): - test/local-embedding-provider.test.ts (3 tests): unavailable-path install hint; pipeline called with dtype:q8 + extractor options + mapped Float32Array result; embedBatch shape. - test/clip-embedding-provider.test.ts (5 tests): unavailable-path; text pipeline dtype:q8 + result; embedBatch; embedImage with data: URL decode; custom model ID propagation. - test/reranker.test.ts: positive-path using vi.doMock + resetModules. Other: - src/huggingface.d.ts deleted (package ships its own types). - src/xenova.d.ts removed. - src/providers/embedding/clip.ts: inline single-use DIMENSIONS constant. - tsdown.config.ts: trim neverBundle list and comment. - README.md L1267: BGE-small -> Xenova/all-MiniLM-L6-v2 (was always wrong). - 16 docs: install commands + prose mentions across main README, 11 translations, SECURITY.md, 2 benchmark docs, benchmark script. - Model IDs (Xenova/all-MiniLM-L6-v2, Xenova/clip-vit-base-patch32, Xenova/ms-marco-MiniLM-L-6-v2) kept — HF Hub repo names, still valid. Closes #1095. Fixes #479. Verified: 1424/1424 tests pass, build clean, tsc clean on migrated files. * test(embedding): add v4 smoke test, harden import errors, expand CI matrix Review follow-ups for #1096: - env-guarded non-mocked smoke test (RUN_HF_SMOKE=1) loading real Xenova/all-MiniLM-L6-v2, asserts 384 finite dims; skipped by default - selective ERR_MODULE_NOT_FOUND handling in local/clip providers so real init errors propagate (checks err.code and err.cause.code to handle vitest mock-factory wrapping) - CLIP install hint made embedding-agnostic (loader serves text + image) - afterEach mock cleanup in doMock-based provider/reranker tests - CI Node matrix: [20, 22] -> [20, 22, 24, 26] across ubuntu/macos * refactor(embedding): drop err.cause check, use manual mock for missing-module tests The .cause branch in the ERR_MODULE_NOT_FOUND check existed only to accommodate vitest's mock-factory wrapping, not a real Node loader behavior. Replace it with a manual mock fixture (__mocks__/@huggingface/ transformers.ts) that throws a Node-shaped error at module top-level, bypassing vitest's factory wrapper so the import rejects with err.code set directly. Production code now checks only err.code === 'ERR_MODULE_NOT_FOUND', matching real Node behavior. Tests verify the same public contract without coupling production code to the test framework.
3.5 KiB
3.5 KiB
LongMemEval-S Benchmark Results
LongMemEval (ICLR 2025) is an academic benchmark for evaluating long-term memory in chat assistants. It tests 5 core abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
Setup
- Dataset: LongMemEval-S (500 questions, ~48 sessions per question, ~115K tokens)
- Source: xiaowu0162/longmemeval-cleaned
- Metric:
recall_any@K— does ANY gold session appear in top-K retrieved results? - Embedding model:
all-MiniLM-L6-v2(384 dimensions, local, no API key) - No LLM in the loop: Pure retrieval evaluation, no answer generation or judge
Results
| System | R@5 | R@10 | R@20 | NDCG@10 | MRR |
|---|---|---|---|---|---|
| agentmemory BM25+Vector | 95.2% | 98.6% | 99.4% | 87.9% | 88.2% |
| agentmemory BM25-only | 86.2% | 94.6% | 98.6% | 73.0% | 71.5% |
| MemPalace raw (vector-only) | 96.6% | ~97.6% | — | — | — |
By Question Type (BM25+Vector)
| Type | R@5 | R@10 | Count |
|---|---|---|---|
| knowledge-update | 98.7% | 100.0% | 78 |
| multi-session | 97.7% | 100.0% | 133 |
| single-session-assistant | 96.4% | 98.2% | 56 |
| temporal-reasoning | 95.5% | 97.7% | 133 |
| single-session-user | 90.0% | 97.1% | 70 |
| single-session-preference | 83.3% | 96.7% | 30 |
By Question Type (BM25-only)
| Type | R@5 | R@10 | Count |
|---|---|---|---|
| knowledge-update | 92.3% | 98.7% | 78 |
| single-session-user | 91.4% | 95.7% | 70 |
| temporal-reasoning | 88.0% | 94.7% | 133 |
| multi-session | 86.5% | 96.2% | 133 |
| single-session-assistant | 80.4% | 91.1% | 56 |
| single-session-preference | 60.0% | 80.0% | 30 |
Analysis
-
BM25+Vector (95.2%) nearly matches pure vector search (96.6%) with only a 1.4pp gap. Both use the same embedding model (all-MiniLM-L6-v2).
-
BM25 alone gets 86.2% — keyword search with Porter stemming and synonym expansion is surprisingly effective on conversational data.
-
Adding vectors to BM25 gives +9pp (86.2% → 95.2%), the largest improvement from any single component.
-
Preferences are the hardest category for both BM25 (60%) and hybrid (83.3%). These require understanding implicit/indirect statements.
-
Multi-session and knowledge-update are strongest (97.7%+ hybrid). The hybrid approach excels when facts are distributed across sessions.
-
R@10 reaches 98.6% — nearly all gold sessions are found within the top 10 results.
Important Notes on Methodology
- These are retrieval recall scores, not end-to-end QA accuracy. The official LongMemEval metric is QA accuracy (retrieve + generate answer + GPT-4o judge).
- Systems on the actual LongMemEval QA leaderboard score 60-95% depending on the LLM reader (Oracle GPT-4o gets ~82.4%).
- We do NOT claim these as "LongMemEval scores" — they are retrieval-only evaluations on the LongMemEval-S haystack.
- Each question builds a fresh index from its ~48 sessions, searches with the question text, and checks if gold session IDs appear in results.
Reproducibility
# Download dataset (264 MB)
pip install huggingface_hub
python3 -c "
from huggingface_hub import hf_hub_download
hf_hub_download(repo_id='xiaowu0162/longmemeval-cleaned', filename='longmemeval_s_cleaned.json', repo_type='dataset', local_dir='benchmark/data')
"
# Run BM25-only
npx tsx benchmark/longmemeval-bench.ts bm25
# Run BM25+Vector hybrid (requires @huggingface/transformers)
npx tsx benchmark/longmemeval-bench.ts hybrid