207 Commits

Author SHA1 Message Date
Zhuanglin Zheng 242e39cbb8 perf: batch convert docs to achieve acceleration (#647) 2026-08-24 19:39:37 +08:00
rayx 91a668ede4 feat(tools): support turbo quantizer in core tool set (#684) 2026-08-24 15:37:33 +08:00
lichen2015 8321c1314a ci: add release workflow to publish prebuilt SDK binaries (#628) 2026-08-24 14:29:28 +08:00
Jalin Wang cb03985e3f fix(db): clean up retired segment directories after optimize() on Windows (#676) 2026-08-21 11:48:00 +08:00
Yin Yongqi 1f7c0696af feat(iterator): add DocIterator for full collection traversal (#597)
Add streaming full-collection traversal across C++/C/Python (relates to #380).

## API
- C++: `Collection::create_iterator(IteratorOptions)` → `DocIterator` (`next()` returns `Result<Doc::Ptr>`: error / `nullptr` EOF / doc; `close()`, idempotent, also run by the destructor).
- C API: opaque `zvec_doc_iterator_t` + `zvec_iterator_options_t` handles; `zvec_collection_create_iterator` / `zvec_doc_iterator_next` / `zvec_doc_iterator_close`; errors mapped precisely to `zvec_error_code_t`.
- Python: `collection.iter_docs()` returns a `DocIterator` (iterator protocol + context manager; prefer `with collection.iter_docs() as docs:`). The snapshot is taken at call time; the iterator closes itself when exhausted, on `with` exit, or via `close()`, releasing the native slot on every path (including early break and exceptions).

## Snapshot semantics
- `create_iterator()` seals the current writing segment on writable collections (read-only collections scan directly, including their writing segment — no flush, no data loss); the snapshot captures the segment set, a deep copy of the delete bitmap, and the schema. Writes after creation are invisible to the iterator; deletions after creation do not affect it.
- `IteratorOptions.output_fields_` selects forward fields (unknown/duplicate names rejected with an error); `include_vector_` controls vector materialization.

## Concurrency (admission control + active-iterator count)
- Iterators and maintenance operations are mutually exclusive via `maintenance_mtx_`: `create_iterator()` fails fast with `FailedPrecondition` while a maintenance operation (optimize, schema DDL, close, destroy) is running.
- While any iterator is open (active-iterator count under the schema lock): schema DDL (create/drop index, add/alter/drop column), destroy and close return `FailedPrecondition`; the destructor path instead logs and waits for open iterators (it cannot report errors); optimize fails at its start. flush, writes and queries are not affected; Stats/Schema/Options (shared lock) are not affected.
- The collection must outlive its iterators: close every iterator before closing/releasing the collection (documented in the C++/C API headers).

## Implementation notes
- Per-segment readers opened lazily (at most one open at a time); deleted rows filtered by `FilteringReader`, wrapped only when the snapshot's delete bitmap is non-empty (`src/db/index/segment/filtering_reader.*`).
- Each batch is materialized column by column in bounded windows of at most 4096 rows (`kMaxRecordBatchNumRows`): a Parquet scan returns a whole row group per ReadNext (up to ~1M rows), so windows cap doc materialization and keep memory constant. Column indices are resolved and validated once per segment reader; scalars/arrays go through the shared column-level converter; vectors are fetched per field using segment-local row ids (`_zvec_row_id_`, correct for compacted segments). Materialization and reader failures are sticky (the error keeps being returned; no partial docs are ever handed out).
- Shared converters in `src/db/index/common/doc_field_converter.*` also serve the SQL engine (`ConvertVectorDataBufferToDocField` / `ConvertArrowColumnToDocFields`; `SegmentImpl::Fetch` keeps its pre-existing implementation).
- Known limitation: docs without a value for a vector field are accepted by insert (the write path only warns and skips) and are indistinguishable from fetch errors today, so iteration with `include_vector` fails on such docs — documented in code with a TODO; tracked in a separate issue.

## Tests
- C++ iterator_test: 22 tests — basic/empty/deleted/close-then-next, include/exclude vector, output_fields selection + rejection, scalar type mapping, 1000-doc integration, read-only collection, performance (100k docs, constant memory), Parquet large-row-group windowed materialization, and concurrency: snapshot isolation under writes, optimize/DDL/close/destroy rejected while open (recovered after the iterator closes), create-iterator rejected while optimize runs, the destructor waits for open iterators, create-iterator-on-closed-collection rejected, slot released by destructor, multiple iterators.
- C c_api_test: 6 iterator tests (basic/concurrent-semantics incl. destroy rejection + FAILED_PRECONDITION mapping, exclude-vector, output-fields, empty, null-args) inside the 75-test C API suite.
- Python test_iter_docs.py: 11 tests (basic fields/vectors, deletion filtering, output fields, isolation, snapshot-at-call-time, iterator protocol, context manager, early close releases the slot).
2026-08-21 10:12:01 +08:00
lichen2015 ae9e61df30 feat: slim down zvec dynamic libraries (#627) 2026-08-20 08:01:06 +08:00
Yin Yongqi ce02c83b19 fix(db): remove orphaned segment directories during crash recovery (#674) 2026-08-18 16:20:03 +08:00
egolearner 5b2f5b409d refactor: use snake_case for public C++ APIs (#683) 2026-08-18 16:14:26 +08:00
Zhuanglin Zheng f0857c8b6e fix: fix pq_int8 bug on neon (#682) 2026-08-17 20:35:38 +08:00
rayx 26c8e7504d feat(diskann): add Linux ARM64 and macOS ARM64 support (#557)
Co-authored-by: Zefeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-08-14 15:07:06 +08:00
Cuiys 1bc05e8e3d fix(sqlengine): isolate equal-or rewrite across AND (#679) 2026-08-13 22:45:22 +08:00
lichen2015 2adb488a28 fix(c-api): stop reporting bogus 0.2.1 version when git tags are missing (#629) 2026-08-13 20:37:16 +08:00
Jalin Wang ba31087c7b feat(turbo): add int8/int4/fp16 record quantizers with scalar kernels, rework kernel dispatch (#624) 2026-08-13 19:54:42 +08:00
luoxiaojian 0ea9003087 feat(vamana): add optional two-pass graph build (#634) 2026-08-13 17:01:07 +08:00
Zhuanglin Zheng c93d4ba4fb feat(PQ-INT8): Added PQ INT8 quantizer in Turbo (#554)
Co-authored-by: ray <rui.xing@alibaba-inc.com>
2026-08-13 13:42:10 +09:00
Jalin Wang b599cbb4d7 fix: asan issues(visit_filter off-by-one and other) (#650) 2026-08-12 19:17:16 +08:00
luoxiaojian 97de47d454 feat(quantization): expose uniform uint7 and add uint8 (#608) 2026-08-12 19:06:58 +08:00
egolearner 8d15e6505b fix(kmeans): correct spherical K-MC2 sampling weights (#655) 2026-08-12 11:44:02 +08:00
Jalin Wang 5fbdd48cb7 fix(core - heap): prevent bypassing Heap invariants (#633) 2026-08-12 11:30:39 +08:00
rayx 82eb4374eb feat(hnsw): support building the graph from original vectors via provider (#657) 2026-08-11 17:18:31 +08:00
Qinren Zhou 4026042c50 fix: handle UTF-8 paths correctly on Windows (#666)
Co-authored-by: Jalin Wang <wangjianning.wjn@alibaba-inc.com>
2026-08-11 17:04:37 +08:00
egolearner 93accc4dac feat: support ivf rabitq (#540) 2026-08-11 15:19:13 +08:00
JinHyuk Sung 478e93e27a fix(python): add collection close method (#567) 2026-08-11 14:17:44 +08:00
rayx f7b595c474 perf(diskann): async I/O overlap and dynamic beam (#617) 2026-08-11 10:18:26 +08:00
egolearner 9b1c037e21 fix(db): handle mismatched scalar batch boundaries (#639) 2026-08-11 10:05:57 +08:00
egolearner 630da45cbf feat(rabitq): enable runtime AVX2 and AVX512 dispatch (#632) 2026-08-10 16:45:26 +08:00
suyanpanghuang 6acf0d926b feat: support linux with musl libc (#649) 2026-08-10 14:02:33 +08:00
rayx 3b1a10c38d feat(diskann): io uring backend (#599)
Co-authored-by: ZeFeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-08-07 17:44:58 +08:00
ZeFeng Yin e1f11e29fe fix(search): validate filters and reset stale filter state (#663) 2026-08-07 16:14:44 +08:00
Yin Yongqi 16c14dc64b fix(collection): allow writes and reads to proceed during Optimize (#614) 2026-08-05 14:46:42 +08:00
egolearner d7a95d02be fix(kmeans): correct spherical centroid normalization (#654) 2026-08-05 12:48:51 +08:00
ZeFeng Yin 9ff5db33db refactor: remove redundant IndexLogger (#652) 2026-08-05 10:51:18 +08:00
egolearner 6eed986ac5 fix: properly handle larger final IPC chunk in mmap store (#645) 2026-08-04 16:58:10 +08:00
egolearner 58375ff7b8 feat(fts): add ngram tokenizer (#593) 2026-08-03 14:06:15 +08:00
luoxiaojian ad36b75966 fix(vamana): honor asymmetric query metrics (#635) 2026-07-30 20:11:37 +08:00
rayx d15a37e425 fix: remove redundant virtual (#631) 2026-07-29 19:23:16 +08:00
luoxiaojian 1ad6df2539 refactor(thread-pool): make CPU affinity opt-in (#623) 2026-07-29 15:30:04 +08:00
egolearner 31be25598a fix(db): persist delete-only writing segment records (#618) 2026-07-28 20:36:02 +08:00
feihongxu0824 e2ea49d4ac fix: limit Windows all-in-one DLL exports (#611) 2026-07-28 11:51:44 +08:00
ZeFeng Yin 8f276e07b1 fix(diskann): make Linux AIO fallback safe for partial operations (#615) 2026-07-27 16:57:52 +08:00
Zhuanglin Zheng a4347e31f1 Add a preprocessor in Turbo (#548)
Co-authored-by: ray <rui.xing@alibaba-inc.com>
2026-07-24 14:01:52 +08:00
ZeFeng Yin fea2481342 fix: HnswExternalStreamerEntity (#607) 2026-07-20 17:29:23 +08:00
rayx cb42297298 fix(diskann): add diskann into zvec core lib (#602) 2026-07-19 12:03:35 +08:00
feihongxu0824 e89be98bb3 fix: optimize after crash recovery without opening wal (#600) 2026-07-17 16:56:21 +08:00
mrcs64 a5dfec6a65 fix(segment): propagate forward-store writer open failure instead of crashing (#579)
Co-authored-by: mrcs64 <9069178+mrcs64@users.noreply.github.com>
2026-07-17 16:13:50 +08:00
rayx ec8a78ee08 refactor(diskann): decouple from libaio via dlopen (#532)
Co-authored-by: Zefeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-07-16 17:58:25 +08:00
egolearner 23538ab876 fix(collection): preserve writing segment on DDL failure (#574) 2026-07-16 10:54:56 +08:00
rayx 8693cf9de7 refactor: turbo quantizer (#546) 2026-07-15 19:54:07 +08:00
ZeFeng Yin f9ef15a362 chore: some optimize in ivf index (#528) 2026-07-14 10:55:51 +08:00
Jalin Wang 1afdea8dc5 feat(python): expose group-by search to Python API (#561)
Co-authored-by: jiliang.ljl <jiliang.ljl@alibaba-inc.com>
2026-07-13 20:22:07 +08:00