build(gomlx): provide libtokenizers for the XLA build's rust tokenizer

The embeddings_gomlx XLA build failed to link on a clean Linux runner with
'cannot find -ltokenizers': hugot's XLA session uses the rust tokenizer, which
is statically linked as libtokenizers.a — a native dependency the earlier
change did not account for (it happened to be present on the dev machine). The
pure-Go default and gortex's own ONNX path do not need it.

Install the matching daulet/tokenizers v1.27.0 archive in the CI gomlx step
(mirroring the onnxruntime install), teach 'make build-gomlx' to fetch it
cross-platform, and document the requirement so a user building the XLA backend
knows to provide libtokenizers.a.

.github/workflows/ci.yml

@@ -90,6 +90,15 @@ jobs:
       - name: Build default (Hugot bundled — no tag)
         run: go build -o gortex-default ./cmd/gortex/
 
+      - name: Install rust tokenizers library
+        # hugot's XLA session uses the rust tokenizer, statically linked as
+        # -ltokenizers, so the embeddings_gomlx XLA build needs libtokenizers.a
+        # on the linker path. (The pure-Go default and the ONNX build do not.)
+        run: |
+          curl -fsSL https://github.com/daulet/tokenizers/releases/download/v1.27.0/libtokenizers.linux-amd64.tar.gz | tar -xz
+          sudo cp libtokenizers.a /usr/lib/
+          sudo cp libtokenizers.a /usr/local/lib/
+
       - name: Build with GoMLX + XLA tags
         run: go build -tags "embeddings_gomlx XLA" -o gortex-gomlx ./cmd/gortex/
 

Makefile

@@ -185,7 +185,23 @@ deps-gomlx:
 	@echo "=== GoMLX dependency ==="
 	go get github.com/gomlx/gomlx@latest
 	go get github.com/gomlx/onnx-gomlx@latest
-	@echo "✓ GoMLX + ONNX converter installed"
+	@echo "=== rust tokenizers (the XLA session links libtokenizers.a statically) ==="
+	@if [ -f /usr/lib/libtokenizers.a ] || [ -f /usr/local/lib/libtokenizers.a ]; then \
+		echo "✓ libtokenizers.a already present"; \
+	else \
+		os=$$(uname -s | tr '[:upper:]' '[:lower:]'); arch=$$(uname -m); \
+		case "$$os-$$arch" in \
+			darwin-arm64) asset=libtokenizers.darwin-arm64.tar.gz; dest=/usr/local/lib ;; \
+			darwin-x86_64) asset=libtokenizers.darwin-x86_64.tar.gz; dest=/usr/local/lib ;; \
+			linux-x86_64|linux-amd64) asset=libtokenizers.linux-amd64.tar.gz; dest=/usr/lib ;; \
+			linux-aarch64|linux-arm64) asset=libtokenizers.linux-arm64.tar.gz; dest=/usr/lib ;; \
+			*) echo "No prebuilt libtokenizers for $$os-$$arch — build it from https://github.com/daulet/tokenizers and place libtokenizers.a on the linker path"; exit 1 ;; \
+		esac; \
+		echo "  downloading $$asset -> $$dest"; \
+		curl -fsSL "https://github.com/daulet/tokenizers/releases/download/v1.27.0/$$asset" | tar -xz -C /tmp; \
+		sudo cp /tmp/libtokenizers.a "$$dest/"; \
+	fi
+	@echo "✓ GoMLX + ONNX converter + rust tokenizers installed"
 	@echo "  Note: XLA/PJRT plugin will auto-download on first run (~100MB)"
 
 # Hugot — uses same XLA/PJRT backend as GoMLX

docs/semantic-search.md

@@ -85,17 +85,17 @@ Opt-in faster local backends via build tags:
 
 ```bash
 go build -tags embeddings_onnx ./cmd/gortex/          # needs: brew install onnxruntime
-go build -tags "embeddings_gomlx XLA" ./cmd/gortex/   # activates the XLA/GoMLX backend; PJRT plugin auto-downloads at runtime
+go build -tags "embeddings_gomlx XLA" ./cmd/gortex/   # needs libtokenizers.a on the linker path — use `make build-gomlx` (see below)
 ```
 
 The `embeddings_onnx` backend (GTE-small) **never auto-downloads**: place `model.onnx` and `vocab.txt` in `~/.gortex/models/gte-small/` yourself and install the ONNX Runtime native library (`brew install onnxruntime`, or the distro equivalent). Without both, the backend reports "ONNX model not found" and the local chain falls through to the pure-Go Hugot backend.
 
-The GoMLX/XLA backend requires **both** tags — `embeddings_gomlx` alone links a disabled XLA stub and always falls through to the pure-Go backend; the `XLA` tag is what compiles the real XLA session. The tag pair is compile-verified, but XLA/PJRT runtime viability is platform-dependent and still experimental — if the plugin fails to load, the local chain degrades to the pure-Go Hugot backend and the startup log names the failed backend (see Troubleshooting). The default pure-Go backend needs no tags and is the reliable path.
+The GoMLX/XLA backend requires **both** tags — `embeddings_gomlx` alone links a disabled XLA stub and always falls through to the pure-Go backend; the `XLA` tag is what compiles the real XLA session. It also statically links the rust tokenizer, so the build needs `libtokenizers.a` on the linker path (a prebuilt archive from [daulet/tokenizers](https://github.com/daulet/tokenizers) releases, at `/usr/lib` or `/usr/local/lib`); `make build-gomlx` downloads it for you. At runtime the XLA/PJRT plugin auto-downloads (~100 MB). XLA/PJRT runtime viability is platform-dependent and still experimental — if the plugin fails to load, the local chain degrades to the pure-Go Hugot backend and the startup log names the failed backend (see Troubleshooting). The default pure-Go backend needs no tags and no native libraries, and is the reliable path.
 
 | Build tag | Backend | Model | Extra dependency | Status |
 |---|---|---|---|---|
 | _(none)_ | Hugot pure-Go | MiniLM-L6-v2 (auto-download) | none | **default — reliable path** |
-| `embeddings_gomlx XLA` | Hugot + XLA/GoMLX | MiniLM-L6-v2 (auto-download) | PJRT plugin (runtime download) | experimental — compile-verified; XLA/PJRT runtime is platform-dependent |
+| `embeddings_gomlx XLA` | Hugot + XLA/GoMLX | MiniLM-L6-v2 (auto-download) | libtokenizers.a (build) + PJRT plugin (runtime download) | experimental — XLA/PJRT runtime is platform-dependent |
 | `embeddings_onnx` | ONNX Runtime | GTE-small (manual placement) | libonnxruntime + hand-placed model | manual setup — never auto-downloads |
 
 The legacy `--embeddings` / `--embeddings-url` / `--embeddings-model` CLI flags and the `GORTEX_EMBEDDINGS*` env vars still take precedence over the config block — useful for one-shot overrides without editing `.gortex.yaml`.