github-featured/ryancodrai--turbovec
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

feat: convert index files between v5, v6 and v7 in any direction (#536)

Browse Source 
* feat: convert index files between v5, v6 and v7 in any direction

The rest of the crate reads and writes v7 only, so bringing an older
file forward needed somewhere for the retired codecs to live. This is
that place, and it goes both ways: any of the three versions in, any of
the three out, for .tv and .tvim.

read() decodes into a version-neutral Image, write() re-encodes it,
convert_file() does both through a temp file and an atomic rename, and
version_of() reports a file's version without decoding it. An example
binary exposes the same from a shell.

Converting is a re-container, not a re-quantize: codes, scales,
calibration and ids cross untouched, which the tests assert directly
(the packed codes must come back byte-identical through all nine version
pairs). v7 output goes through the shipping writer, so it is
byte-identical to what this build would write.

v5 gains a writer it never had — it was read-only in every shipped build
— derived from the reader's layout. Verified the strongest way
available: files this converter writes are loaded by a genuine pre-#535
build's v5/v6 readers, and the search results match the v7 original
exactly, for both .tv and .tvim.

Refused rather than fudged: a lazy index cannot go to v5 or v6, neither
of which can express 'no dimension committed', and pre-v5 files remain
undecodable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: name precisely why v1-v4 cannot be converted

The old message lumped versions 1 through 4 together as 'predates the
v5 rotation change'. The record is more specific, and the distinction
matters to anyone holding such a file.

v1 (turbovec <= 0.4.3) was already refused by the build that introduced
v2 — it has never been decodable by a shipped reader.

v2-v4 are decodable in principle, but only under the pre-v5 rotation: a
QR of a seeded Gaussian, built through a BLAS this crate no longer
depends on, which differed by about one ulp across CPU architectures and
thread counts. That non-reproducibility is why v5 replaced it and why v4
carries a rotation fingerprint at all. Converting one forward is not a
re-container like v5<->v6<->v7: it would mean dequantizing,
inverse-rotating under a rotation this build cannot reliably reproduce,
re-rotating and re-quantizing — lossy, and not guaranteed to be the
rotation that wrote the file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the lazy sentinel is not a v7 invention, and bound the row count

Two review findings on the converter, one of which was me being wrong.

I claimed v5 and v6 had no way to express "no dimension committed" and
refused to write one. They do: validate_header_fields on the previous
release accepts dim == 0 alongside n_vectors == 0, read_v5_header is
shared by both v5 and v6 readers, and that release's own test suite pins
the round trip. So lazily-saved files exist in the wild, and the
converter both refused to read them and refused to produce one — the
opposite of what a converter is for. It now carries the sentinel in
every direction, writing a zeroed codebook for v6 exactly as that
release did.

n_vectors came off the legacy header as an unvalidated u64 and fed every
size calculation: the blocked-length product, packed_row * n_vectors,
rd_f32s's n * 4, and a Vec::with_capacity. Large values wrap, so the
slice bounds checks pass on an empty payload and the allocation is still
reached. Bounded against the file first — a row costs at least five
bytes, so a file this small cannot describe that many rows whatever its
header says. dim is now capped at MAX_DIM on this path too.

Three tests: the sentinel converts in all nine directions and still
loads lazy through the real v7 loader; dim 0 claiming rows is refused by
every version; and a header claiming u64::MAX rows is refused rather
than multiplied out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: point the refusal at the converter, now that it exists

The base PR deliberately worded the docs and legacy_format_error for
what it shipped alone — no converter — and said the follow-up would
update them. This is that follow-up, so a v5 or v6 file is now told it
converts forward rather than that it has to be re-saved by an older
release. Versions 1 through 4 are unchanged: nothing can decode them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: validate a hand-built Image, and finish the sentinel reversal

Four review findings.

convert::write checked scales and ids but not the geometry. The v7 arm
inherits from_parts' validation; the two legacy arms had none, and Image
is public with public fields and no #[non_exhaustive], so a hand-built
one reaches them directly — a short code buffer would be written into a
file no reader can make sense of. Validate bit_width, dim and the packed
length once, before dispatch, for all three versions. The sentinel check
that was inside write_legacy moves there too, so it now covers v7 as
well. Tested per version: valid writes, a short code buffer, bit_width
7, dim 65, and an absurd dim.

Two documentation sites still described the refusal that the later
lazy-sentinel commit reversed within this same PR: Image::dim's doc
("only expressible in v6 and v7") and the CHANGELOG paragraph listing
the sentinel as something a downward conversion loses. Both now say what
the code does — all three versions express it, and the tests loop every
direction.

And the id_map doc comment moved back onto from_v7_load. Adding
from_index_and_ids above it re-orphaned the paragraph describing the
`path` parameter, which from_index_and_ids does not take. This is the
same slip that was fixed in #535; introducing the function here brought
it back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: validate the TQ+ pair, and cover calibration and ids together

The review found the one geometry field write() did not check, and the
mutation gate found three gaps. They turn out to be the same blind spot:
nothing exercised calibration and ids in the same file.

The legacy writers trust the TQ+ pair twice — n_calib comes from the
shift array's length while both arrays are emitted — so a pair of
unequal length writes a header saying "uncalibrated" followed by dim
stray floats. Those land where the id table starts and come back as ids,
with no error at any stage: the reader's `n_calib != dim` guard passes
vacuously at zero. v7 caught it through from_parts; v5 and v6 wrote it
happily. write() now requires the two arrays to match each other and to
be 0 or dim, for every version.

The missed mutants, each now killed by the test written for it, verified
by applying the mutation:

- `at += n_calib * 4` -> `-=` in read_legacy. The TQ+ trailer and the id
  table are adjacent, so an offset error in one reads the other — but
  the id-mapped matrix was uncalibrated and the calibration matrix had
  no ids, so neither could see it. A calibrated id-mapped index now
  converts in all nine directions, checking ids, both calibration
  arrays, and the codes.
- `bytes.len() < 5` -> `== 5` in detect. Under 5 bytes the slice of the
  first four would panic rather than error; the junk fixture was 8 bytes.
  Now every length from 0 to 4.
- `dim > MAX_DIM` -> `>=` in write. The largest legal index could not be
  written at all. Same boundary as the loader's, now pinned on both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ryan Codrai
2026-08-18 11:25:19 +01:00
committed by GitHub
parent f10b0fe252
commit 9f2c990be1
9 changed files with 1117 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+28
View File
@@ -13,6 +13,34 @@ appears under each surface it touches.
### turbovec — Rust crate
#### Added
- **`turbovec::convert` converts an index file between every format
turbovec has written.** v5, v6 and v7 in any direction, for both `.tv`
and `.tvim`, so a file written by an older build can be brought forward
— or taken back, for a rollback or to reproduce a bug against an older
reader. `read` decodes any of them into a version-neutral `Image`,
`write` re-encodes it as any version, `convert_file` does both through
a temp file and an atomic rename, and `version_of` reports what a file
is without decoding it. `cargo run --example convert -- <in> <out> v6`
is the same thing from a shell.
This is the one place that still understands v5 and v6; everything else
reads and writes v7 only. Converting is a re-container, not a
re-quantize: the stored codes, scales, calibration and ids are carried
across untouched, so search results are identical whatever route a file
took. v7 output goes through the shipping writer, so a converted file
is byte-identical to one this build would have produced.
What does not survive going down a version is v7's incremental state —
the generation, the pending redo ops and the file's sync claim —
because v5 and v6 are flat snapshots with no commit history. The lazy
sentinel does survive: all three versions spell "no dimension
committed" as `dim == 0` with no rows, which is what the release
before v7 wrote for a store saved before its first add. Files older
than v5 remain undecodable and are named as such rather than guessed
at.
#### Changed
- **v7 is the only format turbovec reads or writes.** `write`,
docs/api.md
+2 -2
View File
@@ -302,8 +302,8 @@ Measured by flipping every one of the 32,912 bits of a 4114-byte `.tv` file in t
This is a deliberate scope choice, not an oversight. A save is atomic and a crash mid-write leaves the previous file intact, so the writer cannot leave a torn index behind; what is out of scope is damage that arrives afterwards. If you need to detect that, checksum the file yourself or store it on a filesystem that does.
`n_calib = 0` in the TQ+ trailer means an uncalibrated index; otherwise it equals `dim`. Only v7 is read: a v5 or v6 file is refused with an error naming its version. Those files are still readable by the release that wrote them, so re-saving there is the migration route. Versions 1 through 4 predate the v5 rotation change and cannot be decoded at all — their codes were encoded under a rotation this build cannot reproduce — so they must be rebuilt from the source vectors.
`n_calib = 0` in the TQ+ trailer means an uncalibrated index; otherwise it equals `dim`. Only v7 is read: a v5 or v6 file is refused with an error naming its version, and `turbovec::convert` moves a file between v5, v6 and v7 in either direction (`cargo run --example convert -- <in> <out> v7`). Versions 1 through 4 predate the v5 rotation change and cannot be decoded at all — their codes were encoded under a rotation this build cannot reproduce — so they must be rebuilt from the source vectors.
`dim = 0` in the core header signals a lazy uncommitted index. It is only valid alongside `n_vectors = 0`; on load it produces an index whose `dim` is `None` until the first `add` / `add_with_ids` call.
Both formats carry a magic + version byte and are stable across minor versions. Breaking changes bump the version byte. `write()`, `to_bytes()` and `sync()` all produce v7: `write()` and `to_bytes()` an *unclaimed* snapshot, `sync()` a container it claims and then updates incrementally (see [Incremental saves](#incremental-saves--sync)). v7 files are not readable by earlier turbovec releases, whose loaders reject the version byte rather than misparse it.
Both formats carry a magic + version byte and are stable across minor versions. Breaking changes bump the version byte. `write()`, `to_bytes()` and `sync()` all produce v7: `write()` and `to_bytes()` an *unclaimed* snapshot, `sync()` a container it claims and then updates incrementally (see [Incremental saves](#incremental-saves--sync)). v7 files are not readable by earlier turbovec releases, whose loaders reject the version byte rather than misparse it; `turbovec::convert` writes a v5 or v6 file for one.
turbovec/examples/convert.rs
+66
View File
@@ -0,0 +1,66 @@
//! Convert an index file between format versions.
//!
//! ```text
//! cargo run --example convert -- <input> <output> <v5|v6|v7>
//! cargo run --example convert -- <input> # report the version
//! ```
//!
//! The version of the input is detected; any version converts to any
//! other, for both `.tv` and `.tvim`.
use std::path::Path;
use std::process::ExitCode;
use turbovec::convert::{self, Kind, Version};
fn parse(v: &str) -> Option<Version> {
match v.trim().to_ascii_lowercase().as_str() {
"v5" | "5" => Some(Version::V5),
"v6" | "6" => Some(Version::V6),
"v7" | "7" => Some(Version::V7),
_ => None,
}
}
fn main() -> ExitCode {
let args: Vec<String> = std::env::args().skip(1).collect();
match args.len() {
1 => match convert::version_of(Path::new(&args[0])) {
Ok((v, k)) => {
let kind = match k {
Kind::Plain => "positional (.tv)",
Kind::IdMapped => "id-mapped (.tvim)",
};
println!("{}: {v} {kind}", args[0]);
ExitCode::SUCCESS
}
Err(e) => {
eprintln!("{}: {e}", args[0]);
ExitCode::FAILURE
}
},
3 => {
let Some(to) = parse(&args[2]) else {
eprintln!("unknown target version {:?} (expected v5, v6 or v7)", args[2]);
return ExitCode::FAILURE;
};
let (src, dst) = (Path::new(&args[0]), Path::new(&args[1]));
let from = convert::version_of(src);
match convert::convert_file(src, dst, to) {
Ok(()) => {
let from = from.map(|(v, _)| v.to_string()).unwrap_or_default();
println!("{} ({from}) -> {} ({to})", args[0], args[1]);
ExitCode::SUCCESS
}
Err(e) => {
eprintln!("converting {} -> {}: {e}", args[0], args[1]);
ExitCode::FAILURE
}
}
}
_ => {
eprintln!("usage: convert <input> [<output> <v5|v6|v7>]");
ExitCode::FAILURE
}
}
}
turbovec/src/convert.rs
+552
View File
@@ -0,0 +1,552 @@
//! Convert an index file between every format turbovec has written.
//!
//! The rest of the crate reads and writes v7 only. This module is the
//! one place that still understands v5 and v6, and it exists so an index
//! written by an older build can be brought forward — or, for a rollback
//! or a bug report against an older reader, taken back.
//!
//! Both directions between all three versions are supported, for both
//! `.tv` (positional) and `.tvim` (id-mapped) files, so an input in any
//! version can be written out in any version.
//!
//! # The formats
//!
//! All three start `TVPI` or `TVIM` (four bytes) then a version byte.
//!
//! * **v5** — `bit_width` u8, `dim` u32, `n_vectors` u64, then the
//! bit-plane *packed* rows, then one f32 scale per row, then the TQ+
//! trailer (`n_calib` u32, then `n_calib` shifts and `n_calib`
//! scales). No codebook: it is derived from `(bit_width, dim)`.
//! * **v6** — the same header, then the codebook (`2^bits - 1`
//! boundaries and `2^bits` centroids) so a load can verify it has not
//! drifted, then the codes in the arch-neutral *sequential blocked*
//! layout (padded to whole 32-row blocks), then scales, then the same
//! trailer.
//! * **v7** — the container `sync` maintains: a superblock, two
//! alternating header slots, then whole block units. Written here
//! through the shipping writer rather than by hand, so a converted
//! file is byte-identical to one this build would have produced.
//!
//! `.tvim` appends the id table — one u64 per row — after the core in
//! v5 and v6; in v7 the ids ride inside the block units.
//!
//! # What is preserved
//!
//! The codes, the per-row scales, the TQ+ calibration and the ids. The
//! quantized bytes are never re-encoded — converting is a re-container,
//! not a re-quantize, so a round-trip through any version returns the
//! same search results.
//!
//! What cannot survive a conversion *down*: v7's incremental state. A
//! v5 or v6 file is a flat snapshot with no commit history, so the
//! generation, the pending redo ops and the file's claim are dropped.
//! Converting back up produces an unclaimed snapshot, exactly as
//! [`crate::TurboQuantIndex::write`] does.
//!
//! The lazy sentinel *does* survive in both directions: all three
//! versions spell "no dimension committed yet" as `dim == 0` with no
//! rows, and the release before v7 wrote exactly that for a store saved
//! before its first add.
use std::io;
use std::path::Path;
use crate::{codebook, pack, BLOCK};
/// A format version of a turbovec index file.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Version {
/// Packed rows, no embedded codebook. Read-only in every shipped
/// build since v6; this module can also write it.
V5,
/// Sequential-blocked codes with an embedded codebook.
V6,
/// The sync container. What this build reads and writes natively.
V7,
}
impl Version {
fn byte(self) -> u8 {
match self {
Version::V5 => 5,
Version::V6 => 6,
Version::V7 => 1, // v7's own revision byte; the magic distinguishes it
}
}
}
impl std::fmt::Display for Version {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(match self {
Version::V5 => "v5",
Version::V6 => "v6",
Version::V7 => "v7",
})
}
}
/// Which index type a file holds.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Kind {
/// `.tv` — positional, results are slot indices.
Plain,
/// `.tvim` — carries an external id per row.
IdMapped,
}
/// The version-neutral contents of an index file.
///
/// Codes are always the canonical bit-plane *packed* rows here, whatever
/// the file stored, so every writer starts from one representation.
#[derive(Debug, Clone, PartialEq)]
pub struct Image {
pub bit_width: usize,
/// `0` for a lazy index that never committed a dimension (only legal
/// with `n_vectors == 0`; all three versions can express it).
pub dim: usize,
pub n_vectors: usize,
pub packed_codes: Vec<u8>,
pub scales: Vec<f32>,
pub tqplus_shift: Vec<f32>,
pub tqplus_scale: Vec<f32>,
/// Present exactly when the file is `.tvim`.
pub ids: Option<Vec<u64>>,
}
impl Image {
fn kind(&self) -> Kind {
if self.ids.is_some() {
Kind::IdMapped
} else {
Kind::Plain
}
}
}
const TV_MAGIC: &[u8; 4] = b"TVPI";
const TVIM_MAGIC: &[u8; 4] = b"TVIM";
const V5_HEADER: usize = 13;
fn bad(msg: impl Into<String>) -> io::Error {
io::Error::new(io::ErrorKind::InvalidData, msg.into())
}
/// The version and kind of `bytes`, without decoding it.
pub fn detect(bytes: &[u8]) -> io::Result<(Version, Kind)> {
if bytes.len() < 5 {
return Err(bad("too short to be an index file"));
}
if &bytes[0..4] == crate::io_v7::V7_MAGIC {
// v7 carries the kind in the superblock rather than the magic.
let kind = match bytes.get(6) {
Some(0) => Kind::Plain,
Some(1) => Kind::IdMapped,
other => return Err(bad(format!("unknown v7 index kind {other:?}"))),
};
return Ok((Version::V7, kind));
}
let kind = if &bytes[0..4] == TV_MAGIC {
Kind::Plain
} else if &bytes[0..4] == TVIM_MAGIC {
Kind::IdMapped
} else {
return Err(bad("not a turbovec index (unrecognised magic)"));
};
match bytes[4] {
5 => Ok((Version::V5, kind)),
6 => Ok((Version::V6, kind)),
1 => Err(bad(
"version 1 (turbovec <= 0.4.3) was already refused by the build \
that introduced version 2; it cannot be decoded and must be \
rebuilt from the source vectors",
)),
v @ 2..=4 => Err(bad(format!(
"version {v} stores codes encoded under the pre-v5 rotation — a QR \
of a seeded Gaussian, built with a BLAS this crate no longer \
depends on and which differed by ~1 ulp across CPU architectures \
and thread counts (the reason v5 replaced it, and the reason v4 \
carries a rotation fingerprint at all). Those codes cannot be \
re-containered into v5+: they would have to be dequantized, \
inverse-rotated under a rotation this build cannot reliably \
reproduce, re-rotated and re-quantized, which loses accuracy and \
is not guaranteed to be the rotation that wrote them. Rebuild \
from the source vectors instead"
))),
v => Err(bad(format!("unknown index format version {v}"))),
}
}
// ---------------------------------------------------------------------
// Reading
// ---------------------------------------------------------------------
fn rd_u32(b: &[u8], at: usize) -> io::Result<u32> {
b.get(at..at + 4)
.map(|s| u32::from_le_bytes(s.try_into().expect("4 bytes")))
.ok_or_else(|| bad("truncated file"))
}
fn rd_u64(b: &[u8], at: usize) -> io::Result<u64> {
b.get(at..at + 8)
.map(|s| u64::from_le_bytes(s.try_into().expect("8 bytes")))
.ok_or_else(|| bad("truncated file"))
}
fn rd_f32s(b: &[u8], at: usize, n: usize) -> io::Result<Vec<f32>> {
let end = at.checked_add(n * 4).ok_or_else(|| bad("length overflow"))?;
let s = b.get(at..end).ok_or_else(|| bad("truncated file"))?;
Ok(s.chunks_exact(4)
.map(|c| f32::from_le_bytes(c.try_into().expect("4 bytes")))
.collect())
}
/// Rows per packed row, and the sequential-blocked payload length.
fn geometry(bit_width: usize, dim: usize, n: usize) -> (usize, usize) {
let packed_row = dim * bit_width / 8;
let blocked = n.div_ceil(BLOCK) * BLOCK * (dim / (8 / bit_width));
(packed_row, blocked)
}
/// Decode any supported file into the neutral [`Image`].
pub fn read(bytes: &[u8]) -> io::Result<Image> {
let (version, kind) = detect(bytes)?;
match version {
Version::V7 => read_v7(bytes, kind),
Version::V5 | Version::V6 => read_legacy(bytes, version, kind),
}
}
fn read_v7(bytes: &[u8], kind: Kind) -> io::Result<Image> {
let expect_kind = if kind == Kind::IdMapped { 1 } else { 0 };
let mut l = crate::io_v7::load_image(bytes.to_vec(), 0, expect_kind, "the image")?;
let ids = (kind == Kind::IdMapped).then(|| std::mem::take(&mut l.ids));
// The units hold the sequential-blocked layout; the packed rows are
// the neutral form every writer here starts from.
let packed_codes = if l.n_vectors == 0 {
Vec::new()
} else {
let (_, nbg, _) = pack::blocked_geometry(l.n_vectors, l.bit_width, l.dim);
let _ = nbg;
pack::seq_to_packed(&l.seq_blocked, l.n_vectors, l.bit_width, l.dim)
};
Ok(Image {
bit_width: l.bit_width,
dim: l.dim,
n_vectors: l.n_vectors,
packed_codes,
scales: l.scales,
tqplus_shift: l.tqplus_shift,
tqplus_scale: l.tqplus_scale,
ids,
})
}
fn read_legacy(bytes: &[u8], version: Version, kind: Kind) -> io::Result<Image> {
let mut at = 5;
let hdr = bytes
.get(at..at + V5_HEADER)
.ok_or_else(|| bad("truncated header"))?;
let bit_width = hdr[0] as usize;
let dim = u32::from_le_bytes(hdr[1..5].try_into().expect("4 bytes")) as usize;
let n_vectors = usize::try_from(u64::from_le_bytes(
hdr[5..13].try_into().expect("8 bytes"),
))
.map_err(|_| bad("n_vectors does not fit this platform's usize"))?;
at += V5_HEADER;
if !(2..=4).contains(&bit_width) {
return Err(bad(format!("invalid bit_width {bit_width}")));
}
// `dim == 0` is the lazy sentinel — an index that never committed a
// dimension — and v5/v6 carry it exactly as v7 does, valid only with
// no rows. Files like that exist: the previous release's `write()`
// emitted one for a store saved before its first add.
if dim == 0 {
if n_vectors != 0 {
return Err(bad(format!(
"dim 0 with {n_vectors} rows: no dimension committed"
)));
}
} else if !dim.is_multiple_of(8) || dim > crate::MAX_DIM {
return Err(bad(format!("invalid dim {dim}")));
}
// `n_vectors` is an untrusted u64 straight off the header, and every
// size below multiplies it. Bound it by what the file could hold
// before any of that arithmetic runs: each row costs at least one
// byte of codes and four of scale, so a file this small cannot
// describe that many rows however its header reads.
if n_vectors.saturating_mul(5) > bytes.len() {
return Err(bad(format!(
"header claims {n_vectors} rows, which a {}-byte file cannot hold",
bytes.len()
)));
}
let (packed_row, blocked_len) = geometry(bit_width, dim, n_vectors);
let packed_codes = match version {
Version::V5 => {
let n = packed_row * n_vectors;
let s = bytes.get(at..at + n).ok_or_else(|| bad("truncated codes"))?;
at += n;
s.to_vec()
}
Version::V6 => {
// The embedded codebook is skipped: it is a function of
// (bit_width, dim) and every writer here re-derives it, so
// carrying the file's copy forward would only let a drifted
// one propagate.
let n_levels = 1usize << bit_width;
at += (2 * n_levels - 1) * 4;
let s = bytes
.get(at..at + blocked_len)
.ok_or_else(|| bad("truncated codes"))?;
at += blocked_len;
if n_vectors == 0 {
Vec::new()
} else {
pack::seq_to_packed(s, n_vectors, bit_width, dim)
}
}
Version::V7 => unreachable!("handled by read_v7"),
};
let scales = rd_f32s(bytes, at, n_vectors)?;
at += n_vectors * 4;
let n_calib = rd_u32(bytes, at)? as usize;
at += 4;
if n_calib != 0 && n_calib != dim {
return Err(bad(format!("calibration length {n_calib} != dim {dim}")));
}
let tqplus_shift = rd_f32s(bytes, at, n_calib)?;
at += n_calib * 4;
let tqplus_scale = rd_f32s(bytes, at, n_calib)?;
at += n_calib * 4;
let ids = match kind {
Kind::Plain => None,
Kind::IdMapped => {
let mut v = Vec::with_capacity(n_vectors);
for i in 0..n_vectors {
v.push(rd_u64(bytes, at + i * 8)?);
}
Some(v)
}
};
Ok(Image {
bit_width,
dim,
n_vectors,
packed_codes,
scales,
tqplus_shift,
tqplus_scale,
ids,
})
}
// ---------------------------------------------------------------------
// Writing
// ---------------------------------------------------------------------
/// Encode an [`Image`] in `version`. The kind follows the image's ids.
pub fn write(image: &Image, version: Version) -> io::Result<Vec<u8>> {
// `Image` is public with public fields, so a caller can hand us one
// that never came from `read`. The v7 arm inherits `from_parts`'
// checks; the legacy arms have none of their own, so validate the
// geometry here for all three rather than trusting the shape.
if !(2..=4).contains(&image.bit_width) {
return Err(bad(format!("invalid bit_width {}", image.bit_width)));
}
if image.dim == 0 {
if image.n_vectors != 0 {
return Err(bad(format!(
"dim 0 is the lazy sentinel and cannot carry {} rows",
image.n_vectors
)));
}
} else if !image.dim.is_multiple_of(8) || image.dim > crate::MAX_DIM {
return Err(bad(format!("invalid dim {}", image.dim)));
}
let expect_codes = image
.n_vectors
.saturating_mul(image.dim)
.saturating_mul(image.bit_width)
/ 8;
if image.packed_codes.len() != expect_codes {
return Err(bad(format!(
"packed_codes is {} bytes, but {} rows at dim {} and {} bits need {}",
image.packed_codes.len(),
image.n_vectors,
image.dim,
image.bit_width,
expect_codes
)));
}
// The TQ+ pair is geometry too, and the legacy writers trust it
// twice over: `n_calib` is taken from the shift array's length while
// *both* arrays are emitted. A mismatched pair therefore writes a
// header saying "no calibration" followed by `dim` stray floats,
// which land where the id table starts and decode as ids — silently,
// since the reader's `n_calib != dim` guard passes vacuously at 0.
let calib = image.tqplus_shift.len();
if calib != image.tqplus_scale.len() {
return Err(bad(format!(
"tqplus_shift has {} entries but tqplus_scale has {}",
calib,
image.tqplus_scale.len()
)));
}
if calib != 0 && calib != image.dim {
return Err(bad(format!(
"calibration length {calib} must be 0 or dim {}",
image.dim
)));
}
if image.scales.len() != image.n_vectors {
return Err(bad(format!(
"{} scales for {} rows",
image.scales.len(),
image.n_vectors
)));
}
if let Some(ids) = &image.ids {
if ids.len() != image.n_vectors {
return Err(bad(format!(
"{} ids for {} rows",
ids.len(),
image.n_vectors
)));
}
}
match version {
Version::V7 => write_v7(image),
Version::V5 | Version::V6 => write_legacy(image, version),
}
}
fn write_v7(image: &Image) -> io::Result<Vec<u8>> {
// Through the shipping writer, so a converted file is byte-identical
// to one this build would have produced from the same index.
let dim = (image.dim != 0).then_some(image.dim);
let inner = crate::TurboQuantIndex::from_parts(
dim,
image.bit_width,
image.n_vectors,
image.packed_codes.clone(),
image.scales.clone(),
image.tqplus_shift.clone(),
image.tqplus_scale.clone(),
)
.map_err(|e| bad(e.to_string()))?;
match &image.ids {
None => Ok(inner.to_bytes()),
Some(ids) => {
let m = crate::IdMapIndex::from_index_and_ids(inner, ids.clone())
.map_err(|e| bad(e.to_string()))?;
Ok(m.to_bytes())
}
}
}
fn write_legacy(image: &Image, version: Version) -> io::Result<Vec<u8>> {
let (_, blocked_len) = geometry(image.bit_width, image.dim, image.n_vectors);
let mut out = Vec::new();
out.extend_from_slice(match image.kind() {
Kind::Plain => TV_MAGIC,
Kind::IdMapped => TVIM_MAGIC,
});
out.push(version.byte());
out.push(image.bit_width as u8);
out.extend_from_slice(&(image.dim as u32).to_le_bytes());
out.extend_from_slice(&(image.n_vectors as u64).to_le_bytes());
if version == Version::V6 {
// Re-derived, never copied from the source file: the codebook is
// a pure function of (bit_width, dim), and a v6 load verifies it
// against exactly this. The lazy sentinel has no dimension to
// solve for, so it embeds zeros — which is what the previous
// release wrote, and what its loader skips validating.
let n_levels = 1usize << image.bit_width;
let (boundaries, centroids) = if image.dim == 0 {
(vec![0.0f32; n_levels - 1], vec![0.0f32; n_levels])
} else {
codebook::codebook(image.bit_width, image.dim)
};
for v in boundaries.iter().chain(centroids.iter()) {
out.extend_from_slice(&v.to_le_bytes());
}
if image.n_vectors == 0 {
out.extend_from_slice(&vec![0u8; blocked_len]);
} else {
let (blocked, _) = pack::repack(
&image.packed_codes,
image.n_vectors,
image.bit_width,
image.dim,
);
let (_, nbg, _) =
pack::blocked_geometry(image.n_vectors, image.bit_width, image.dim);
let seq = pack::native_to_seq(&blocked, image.bit_width, nbg);
debug_assert_eq!(seq.len(), blocked_len);
out.extend_from_slice(&seq);
}
} else {
out.extend_from_slice(&image.packed_codes);
}
for &s in &image.scales {
out.extend_from_slice(&s.to_le_bytes());
}
out.extend_from_slice(&(image.tqplus_shift.len() as u32).to_le_bytes());
for v in image.tqplus_shift.iter().chain(image.tqplus_scale.iter()) {
out.extend_from_slice(&v.to_le_bytes());
}
if let Some(ids) = &image.ids {
for &id in ids {
out.extend_from_slice(&id.to_le_bytes());
}
}
Ok(out)
}
// ---------------------------------------------------------------------
// Files
// ---------------------------------------------------------------------
/// Read `src`, and write it to `dst` in `to`.
///
/// `dst` is written through a temp file and renamed, so a failure leaves
/// any previous file at that path intact.
pub fn convert_file(src: &Path, dst: &Path, to: Version) -> io::Result<()> {
let bytes = std::fs::read(src)?;
let image = read(&bytes)?;
let out = write(&image, to)?;
let (file, tmp) = crate::io::create_tmp(dst)?;
let result = (|| {
use std::io::Write as _;
let mut w = std::io::BufWriter::with_capacity(1 << 20, &file);
w.write_all(&out)?;
w.flush()?;
drop(w);
file.sync_all()
})();
if let Err(e) = result.and_then(|()| {
drop(file);
crate::io::rename_atomic(&tmp, dst)
}) {
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
crate::io::sync_parent_dir_after_commit(dst);
Ok(())
}
/// The version of the file at `path`.
pub fn version_of(path: &Path) -> io::Result<(Version, Kind)> {
let mut head = [0u8; 8];
let f = std::fs::File::open(path)?;
crate::io::read_exact_at(&f, &mut head, 0)?;
detect(&head)
}
turbovec/src/id_map.rs
+33
View File
@@ -890,6 +890,39 @@ impl IdMapIndex {
Self::from_v7_load(l, bind)
}
/// Wrap an already-built index with an id table.
///
/// Used by [`crate::convert`], which decodes a file into codes plus
/// ids and needs to re-emit it: the ids are validated for duplicates
/// exactly as a load does, since a table with a repeat cannot answer
/// `remove` or `contains` unambiguously.
pub(crate) fn from_index_and_ids(
inner: TurboQuantIndex,
slot_to_id: Vec<u64>,
) -> std::io::Result<Self> {
if slot_to_id.len() != inner.len() {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
format!("{} ids for {} rows", slot_to_id.len(), inner.len()),
));
}
let mut sorted = slot_to_id.clone();
sorted.sort_unstable();
if sorted.windows(2).any(|w| w[0] == w[1]) {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"duplicate ids",
));
}
Ok(Self {
inner,
slot_to_id,
id_to_slot: std::sync::OnceLock::new(),
sorted_ids: std::sync::Mutex::new(sorted),
deferred_added: std::sync::Mutex::new(Default::default()),
})
}
/// Shared tail of the path and byte v7 loaders. `path` is `None` for
/// a byte image, which is not a sync destination.
fn from_v7_load(mut l: crate::io_v7::V7Load, path: Option<&Path>) -> std::io::Result<Self> {
turbovec/src/io.rs
+6 -7
View File
@@ -35,11 +35,10 @@
//! them.
//!
//! [`legacy_format_error`] is what remains of that history. It names a
//! file it cannot read rather than guessing: a v5 or v6 index is still
//! readable by the release that wrote it, so re-saving there migrates
//! it, while versions 1 through 4 predate the v5 rotation change —
//! which altered every encoded byte — and can only be rebuilt from the
//! source vectors.
//! file it cannot read rather than guessing: a v5 or v6 index converts
//! forward with [`crate::convert`], while versions 1 through 4 predate
//! the v5 rotation change — which altered every encoded byte — and can
//! only be rebuilt from the source vectors.
//!
//! One case it cannot name: a version 1 `.tv` began with a bare
//! `bit_width` byte and carries no magic at all, so it is indistinguishable
@@ -99,8 +98,8 @@ pub(crate) fn legacy_format_error(path: &Path) -> io::Error {
let detail = match version {
Some(v @ 5..=6) => format!(
"is a version {v} turbovec index; this build reads only the v7 \
format. Open it with the turbovec release that wrote it and save \
it again, or rebuild it from the source vectors"
format. Convert it with turbovec::convert (or the `convert` \
example), which reads v5, v6 and v7 and writes any of them"
),
Some(v @ 1..=4) => format!(
"is a version {v} turbovec index, which predates the v5 rotation \
turbovec/src/lib.rs
+1
View File
@@ -64,6 +64,7 @@ pub mod codebook;
pub mod encode;
pub mod error;
pub mod id_map;
pub mod convert;
pub mod io;
mod io_v7;
pub mod pack;
turbovec/tests/convert_versions.rs
+427
View File
@@ -0,0 +1,427 @@
//! Every version to every version, both index kinds.
//!
//! The bar is not "it parses": a conversion must preserve what the file
//! is *for*, so each case compares search results against the original
//! index, and the id-mapped cases compare resolved ids too.
use turbovec::convert::{self, Image, Kind, Version};
use turbovec::{IdMapIndex, TurboQuantIndex};
const ALL: [Version; 3] = [Version::V5, Version::V6, Version::V7];
fn rows(n: usize, dim: usize, seed: u64) -> Vec<f32> {
let mut v = vec![0.0f32; n * dim];
let mut s = seed | 1;
for x in v.iter_mut() {
s ^= s << 13;
s ^= s >> 7;
s ^= s << 17;
*x = ((s >> 40) as f32 / (1u64 << 23) as f32) - 0.5;
}
v
}
fn build(dim: usize, n: usize, calibrated: bool) -> TurboQuantIndex {
let mut idx = TurboQuantIndex::new(dim, 4).unwrap();
if calibrated {
idx.calibrate(&rows(1024, dim, 3)).unwrap();
}
idx.add(&rows(n, dim, 1));
idx
}
/// Convert `bytes` to `to`, then read it back as an index and compare
/// search results with `want`.
fn assert_plain_parity(bytes: &[u8], to: Version, want: &TurboQuantIndex, dim: usize) {
let image = convert::read(bytes).expect("read source");
let out = convert::write(&image, to).unwrap_or_else(|e| panic!("write {to}: {e}"));
let (v, k) = convert::detect(&out).expect("detect output");
assert_eq!(v, to, "output is not {to}");
assert_eq!(k, Kind::Plain);
// Round the output back through the reader: for v7 that is the real
// loader, for v5/v6 it is this module, which is the only thing that
// still reads them.
let back = convert::read(&out).expect("read back");
assert_eq!(back.n_vectors, want.len());
assert_eq!(back.dim, dim);
let idx = TurboQuantIndex::from_parts(
Some(back.dim),
back.bit_width,
back.n_vectors,
back.packed_codes.clone(),
back.scales.clone(),
back.tqplus_shift.clone(),
back.tqplus_scale.clone(),
)
.expect("rebuild index");
let q = rows(4, dim, 99);
let got = idx.search(&q, 10);
let expect = want.search(&q, 10);
assert_eq!(got.indices, expect.indices, "indices differ after -> {to}");
assert_eq!(got.scores, expect.scores, "scores differ after -> {to}");
}
#[test]
fn every_version_converts_to_every_other_for_tv() {
let dim = 64;
let want = build(dim, 200, false);
// One source file per version, produced by the converter itself, so
// the matrix does not depend on having fixtures for retired formats.
let base = convert::read(&want.to_bytes()).unwrap();
for from in ALL {
let src = convert::write(&base, from).unwrap_or_else(|e| panic!("seed {from}: {e}"));
assert_eq!(convert::detect(&src).unwrap().0, from);
for to in ALL {
assert_plain_parity(&src, to, &want, dim);
}
}
}
#[test]
fn every_version_converts_to_every_other_for_tvim() {
let dim = 64;
let n = 200;
let ids: Vec<u64> = (0..n as u64).map(|i| i * 7 + 11).collect();
let mut want = IdMapIndex::new(dim, 4).unwrap();
want.add_with_ids(&rows(n, dim, 1), &ids).unwrap();
let base = convert::read(&want.to_bytes()).unwrap();
assert_eq!(base.ids.as_deref(), Some(&ids[..]));
let q = rows(4, dim, 99);
let expect = want.search(&q, 10);
for from in ALL {
let src = convert::write(&base, from).unwrap_or_else(|e| panic!("seed {from}: {e}"));
assert_eq!(convert::detect(&src).unwrap(), (from, Kind::IdMapped));
for to in ALL {
let image = convert::read(&src).unwrap();
let out = convert::write(&image, to).unwrap();
assert_eq!(convert::detect(&out).unwrap(), (to, Kind::IdMapped));
let back = convert::read(&out).unwrap();
assert_eq!(back.ids.as_deref(), Some(&ids[..]), "{from} -> {to} lost ids");
let inner = TurboQuantIndex::from_parts(
Some(back.dim),
back.bit_width,
back.n_vectors,
back.packed_codes,
back.scales,
back.tqplus_shift,
back.tqplus_scale,
)
.unwrap();
let bytes = convert::write(
&convert::read(&inner.to_bytes()).map(|mut i| {
i.ids = Some(ids.clone());
i
})
.unwrap(),
Version::V7,
)
.unwrap();
let m = IdMapIndex::from_bytes(&bytes).unwrap();
assert_eq!(m.search(&q, 10), expect, "{from} -> {to} changed results");
}
}
}
#[test]
fn calibration_survives_every_conversion() {
let dim = 64;
let want = build(dim, 128, true);
assert!(!want.tqplus_shift().is_empty(), "fixture must be calibrated");
let base = convert::read(&want.to_bytes()).unwrap();
for from in ALL {
let src = convert::write(&base, from).unwrap();
for to in ALL {
let out = convert::write(&convert::read(&src).unwrap(), to).unwrap();
let back = convert::read(&out).unwrap();
assert_eq!(back.tqplus_shift, base.tqplus_shift, "{from} -> {to} shift");
assert_eq!(back.tqplus_scale, base.tqplus_scale, "{from} -> {to} scale");
}
}
}
/// The lazy sentinel is not a v7 invention: v5 and v6 carry `dim == 0`
/// with no rows too, and the previous release's `write()` emitted one
/// for a store saved before its first add. So it has to convert both
/// ways, or those files cannot be brought forward.
#[test]
fn a_lazy_index_converts_in_every_direction() {
let lazy = TurboQuantIndex::new_lazy(4).unwrap();
let base = convert::read(&lazy.to_bytes()).unwrap();
assert_eq!(base.dim, 0);
assert_eq!(base.n_vectors, 0);
for from in ALL {
let src = convert::write(&base, from).unwrap_or_else(|e| panic!("write {from}: {e}"));
assert_eq!(convert::detect(&src).unwrap().0, from);
for to in ALL {
let out = convert::write(&convert::read(&src).unwrap(), to)
.unwrap_or_else(|e| panic!("{from} -> {to}: {e}"));
let back = convert::read(&out).unwrap();
assert_eq!(back.dim, 0, "{from} -> {to} lost the sentinel");
assert_eq!(back.n_vectors, 0);
}
}
// And it still loads as a lazy index through the real v7 loader.
let v7 = convert::write(&base, Version::V7).unwrap();
let idx = TurboQuantIndex::from_bytes(&v7).unwrap();
assert_eq!(idx.dim_opt(), None);
}
/// `dim == 0` is only legal with no rows, whichever version claims it.
#[test]
fn the_sentinel_is_refused_when_it_claims_rows() {
let mut idx = TurboQuantIndex::new(64, 4).unwrap();
idx.add(&rows(32, 64, 2));
let mut image = convert::read(&idx.to_bytes()).unwrap();
image.dim = 0;
for v in ALL {
let e = convert::write(&image, v).expect_err("dim 0 with rows must be refused");
assert!(
e.to_string().contains("sentinel")
|| e.to_string().contains("dim 0")
|| e.to_string().contains("n_vectors=0"),
"unhelpful for {v}: {e}"
);
}
}
/// A header can claim any row count; the reader must bound it against
/// the file before any size arithmetic runs on it.
#[test]
fn an_absurd_row_count_is_refused_not_multiplied_out() {
let mut v6 = b"TVPI".to_vec();
v6.push(6);
v6.push(4); // bit_width
v6.extend_from_slice(&64u32.to_le_bytes()); // dim
v6.extend_from_slice(&u64::MAX.to_le_bytes()); // n_vectors
v6.extend_from_slice(&[0u8; 32]);
let e = convert::read(&v6).expect_err("an absurd row count must be refused");
assert!(
e.to_string().contains("rows"),
"should name the row count: {e}"
);
}
#[test]
fn pre_v5_files_and_junk_are_named_rather_than_guessed() {
let mut v4 = b"TVPI".to_vec();
v4.push(4);
v4.extend_from_slice(&[0u8; 32]);
let e = convert::read(&v4).expect_err("v4 must be refused");
assert!(e.to_string().contains("pre-v5 rotation"), "got: {e}");
assert!(e.to_string().contains("Rebuild"), "got: {e}");
let mut v1 = b"TVPI".to_vec();
v1.push(1);
v1.extend_from_slice(&[0u8; 32]);
let e = convert::read(&v1).expect_err("v1 must be refused");
assert!(e.to_string().contains("0.4.3"), "got: {e}");
let e = convert::read(b"\x7fELF\x02\x01\x01\x00").expect_err("junk must be refused");
assert!(e.to_string().contains("not a turbovec index"), "got: {e}");
}
#[test]
fn convert_file_writes_atomically_and_detects_the_version() {
let dir = std::env::temp_dir().join(format!("tvconv-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let src = dir.join("in.tv");
let dst = dir.join("out.tv");
let idx = build(64, 96, false);
idx.write(&src).unwrap();
assert_eq!(convert::version_of(&src).unwrap(), (Version::V7, Kind::Plain));
convert::convert_file(&src, &dst, Version::V6).unwrap();
assert_eq!(convert::version_of(&dst).unwrap(), (Version::V6, Kind::Plain));
// No temp left behind.
let strays: Vec<_> = std::fs::read_dir(&dir)
.unwrap()
.filter_map(|e| e.ok())
.map(|e| e.file_name().to_string_lossy().into_owned())
.filter(|n| n.contains(".tmp."))
.collect();
assert!(strays.is_empty(), "temp files left: {strays:?}");
// And back again.
convert::convert_file(&dst, &src, Version::V7).unwrap();
assert_eq!(convert::version_of(&src).unwrap(), (Version::V7, Kind::Plain));
let reloaded = TurboQuantIndex::load(&src).unwrap();
let q = rows(2, 64, 5);
assert_eq!(reloaded.search(&q, 8).indices, idx.search(&q, 8).indices);
std::fs::remove_dir_all(&dir).ok();
}
/// Converting is a re-container, not a re-quantize: the stored codes
/// must come out byte-identical, whatever route they took.
#[test]
fn codes_are_never_re_encoded() {
let dim = 128;
let want = build(dim, 64, true);
let base: Image = convert::read(&want.to_bytes()).unwrap();
for from in ALL {
for to in ALL {
let a = convert::write(&base, from).unwrap();
let b = convert::write(&convert::read(&a).unwrap(), to).unwrap();
let back = convert::read(&b).unwrap();
assert_eq!(
back.packed_codes, base.packed_codes,
"{from} -> {to} changed the stored codes"
);
assert_eq!(back.scales, base.scales, "{from} -> {to} changed scales");
}
}
}
/// `Image` is public with public fields, so it can arrive hand-built
/// rather than from `read`. Every version must check the geometry —
/// the v7 arm inherits `from_parts`' validation, the legacy arms have
/// none of their own, and a short code buffer would otherwise be
/// written straight into a file that no reader can make sense of.
#[test]
fn a_hand_built_image_is_validated_for_every_version() {
let good = |n: usize| Image {
bit_width: 4,
dim: 64,
n_vectors: n,
packed_codes: vec![0u8; n * 64 * 4 / 8],
scales: vec![1.0; n],
tqplus_shift: vec![],
tqplus_scale: vec![],
ids: None,
};
for v in ALL {
assert!(convert::write(&good(2), v).is_ok(), "{v}: the valid case must write");
// A row is 32 bytes at dim 64, 4-bit; supply one.
let mut short = good(1);
short.packed_codes = vec![0u8; 1];
let e = convert::write(&short, v).expect_err("short codes must be refused");
assert!(e.to_string().contains("packed_codes"), "{v}: {e}");
let mut bad_bits = good(1);
bad_bits.bit_width = 7;
assert!(
convert::write(&bad_bits, v).is_err(),
"{v}: bit_width 7 must be refused"
);
let mut bad_dim = good(1);
bad_dim.dim = 65;
assert!(convert::write(&bad_dim, v).is_err(), "{v}: dim 65 must be refused");
let mut huge = good(1);
huge.dim = 1 << 24;
assert!(convert::write(&huge, v).is_err(), "{v}: an absurd dim must be refused");
}
}
/// A mismatched TQ+ pair must not reach the legacy writers.
///
/// They take `n_calib` from the shift array but emit both, so a pair of
/// unequal length writes a header saying "uncalibrated" followed by
/// stray floats — which land where the id table starts and come back as
/// ids, with no error anywhere. v7 caught it through `from_parts`; v5
/// and v6 had nothing.
#[test]
fn a_mismatched_calibration_pair_is_refused_before_it_corrupts_ids() {
let dim = 64;
let n = 2;
let ids: Vec<u64> = vec![111, 222];
let mut img = Image {
bit_width: 4,
dim,
n_vectors: n,
packed_codes: vec![0u8; n * dim * 4 / 8],
scales: vec![1.0; n],
tqplus_shift: vec![],
tqplus_scale: vec![7.5; dim],
ids: Some(ids.clone()),
};
for v in ALL {
let e = convert::write(&img, v).expect_err("a half-empty pair must be refused");
assert!(e.to_string().contains("tqplus"), "{v}: {e}");
}
// A pair of equal but wrong length is refused too.
img.tqplus_shift = vec![0.0; dim / 2];
img.tqplus_scale = vec![1.0; dim / 2];
for v in ALL {
let e = convert::write(&img, v).expect_err("a short pair must be refused");
assert!(e.to_string().contains("calibration length"), "{v}: {e}");
}
}
/// Calibration and ids in the same file, through every version.
///
/// The two features are written adjacently — the TQ+ trailer then the id
/// table — so an offset error in one silently reads the other. Nothing
/// covered both at once: the id-mapped matrix was uncalibrated and the
/// calibration matrix had no ids.
#[test]
fn a_calibrated_id_mapped_index_converts_in_every_direction() {
let dim = 64;
let n = 128;
let ids: Vec<u64> = (0..n as u64).map(|i| i * 5 + 3).collect();
let mut m = IdMapIndex::new(dim, 4).unwrap();
m.calibrate(&rows(1024, dim, 8)).unwrap();
m.add_with_ids(&rows(n, dim, 9), &ids).unwrap();
let base = convert::read(&m.to_bytes()).unwrap();
assert_eq!(base.tqplus_shift.len(), dim, "fixture must be calibrated");
assert_eq!(base.ids.as_deref(), Some(&ids[..]));
for from in ALL {
let src = convert::write(&base, from).unwrap();
for to in ALL {
let out = convert::write(&convert::read(&src).unwrap(), to).unwrap();
let back = convert::read(&out).unwrap();
assert_eq!(back.ids.as_deref(), Some(&ids[..]), "{from} -> {to}: ids");
assert_eq!(back.tqplus_shift, base.tqplus_shift, "{from} -> {to}: shift");
assert_eq!(back.tqplus_scale, base.tqplus_scale, "{from} -> {to}: scale");
assert_eq!(back.packed_codes, base.packed_codes, "{from} -> {to}: codes");
}
}
}
/// Too short to hold a magic and a version byte: an error, not a panic
/// from slicing past the end.
#[test]
fn a_truncated_prefix_is_an_error_not_a_panic() {
for len in 0..5usize {
let bytes = vec![b'T'; len];
let e = convert::detect(&bytes).expect_err("{len} bytes must be refused");
assert!(e.to_string().contains("too short"), "{len}: {e}");
assert!(convert::read(&bytes).is_err());
}
}
/// `dim > MAX_DIM` is the bound; at `>=` the largest legal index cannot
/// be written at all.
#[test]
fn an_image_at_exactly_max_dim_can_be_written() {
let dim = turbovec::MAX_DIM;
let img = Image {
bit_width: 2,
dim,
n_vectors: 0,
packed_codes: Vec::new(),
scales: Vec::new(),
tqplus_shift: Vec::new(),
tqplus_scale: Vec::new(),
ids: None,
};
for v in ALL {
assert!(convert::write(&img, v).is_ok(), "{v}: dim {dim} is legal");
}
}
turbovec/tests/v7_only.rs
+2 -2
View File
@@ -65,8 +65,8 @@ fn a_pre_v7_file_is_refused_with_an_actionable_error() {
assert!(msg.contains("version 6"), "should name the version: {msg}");
assert!(msg.contains("v7"), "should say what is supported: {msg}");
assert!(
msg.contains("save it again") || msg.contains("rebuild"),
"should say what to do about it: {msg}"
msg.contains("convert"),
"a v5/v6 file converts forward, so say so: {msg}"
);
// Versions the v5 rotation change made undecodable get different