github-featured/dmtrkovalenko--fff
2
0
Fork 0
Code Issues Pull Requests Actions 8 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

feat: Improve stability of file search discovery in background (#431)

Browse Source 
* feat: Improve scan runtime performance and RSS

* chore: Comprehensive randomized fuzzy mutation testing

It's such a shame that we keep commiting regressions that are leading ot
the missed events from the file watcher. I want to minimize this to 0.

* chore: Update docs for - chore: Comprehensive randomized fuzzy mutation testing

* chore: Update docs for - chore: Update docs for - chore: Comprehensive randomized fuzzy mutation testing

* test: Verbose stress test output
This commit is contained in:
Dmitriy Kovalenko
2026-05-02 09:28:14 -07:00
committed by GitHub
parent 573a783d2f
commit 1bcbce2bc6
44 changed files with 5181 additions and 1681 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/rust.yml
+55
View File
@@ -19,6 +19,9 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
# Guard against deadlocks in the shared-picker / watcher teardown
# path: a stuck test would otherwise consume a full 6h CI slot.
timeout-minutes: 10
steps:
- uses: actions/checkout@v5
@@ -39,6 +42,58 @@ jobs:
- name: Run tests
run: cargo test --features zlob --workspace --exclude fff-nvim
stress-test:
name: Stress Test (Watcher + Git)
runs-on: ${{ matrix.os }}
strategy:
# Keep going after one OS fails so we can see whether a bug
# reproduces everywhere or is platform-specific.
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
# Long-running; don't let a stuck watcher thread burn a full CI
# timeout. Two scenarios should finish well under this limit.
timeout-minutes: 20
steps:
- uses: actions/checkout@v5
- name: Install Zig
uses: goto-bus-stop/setup-zig@v2
with:
version: 0.16.0
- name: Install Rust
uses: actions-rust-lang/setup-rust-toolchain@v1.15.4
with:
cache: true
cache-on-failure: true
cache-key: "v1-rust-stress-${{ matrix.os }}"
components: rustfmt, clippy
- name: Stress test (seeded / deterministic)
shell: bash
run: make test-stress-seeded
env:
FFF_STRESS_CASES: "3"
FFF_STRESS_MIN_OPS: "30"
FFF_STRESS_MAX_OPS: "50"
- name: Stress test (random / fuzzy)
shell: bash
run: make test-stress-random
env:
FFF_STRESS_CASES: "5"
FFF_STRESS_MIN_OPS: "30"
FFF_STRESS_MAX_OPS: "60"
- name: Upload proptest regressions on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: proptest-regressions-${{ matrix.os }}
path: crates/fff-core/tests/fuzz_git_watcher_stress.proptest-regressions
if-no-files-found: ignore
fmt:
name: cargo fmt
runs-on: ubuntu-latest
.gitignore
+3
View File
@@ -24,3 +24,6 @@ scripts/benchmark-results/
*.dylib
*.so
*.dll
# Instruments traces
*.trace/
AGENTS.md
+110
View File
@@ -0,0 +1,110 @@
# To Clankers
This repository contains **FFF.nvim (Fast File Finder)**, a high-performance file picker for Neovim inspired by blink.cmp's fuzzy matching technology. It's NOT a completion plugin, but rather a standalone file finder with advanced fuzzy search and frecency scoring. The project aims to be the drop-in replacement for telescope, fzf-lua, snacks.picker and similar plugins, focusing on speed, accuracy search and usability features.
## Development Commands
### Building
- `cargo build --release` - Build the Rust fuzzy matcher and file picker
### Testing and Development Tools
This project does not have a traditional test suite. Testing is done through:
- Create e2e local test file for Neovim: Load any Lua test file with `nvim -l <test_file>`
- Write inline rust unit tests for any functionality that is standalone and scoped within a single function
### Code Quality
- `cargo fmt` - Format Rust code (follows standard conventions)
- `make lint` - Rust linting and code analysis
- `make format` - Format all code
- `make test` - Run unit tests (limited coverage, primarily integration testing)
When doing code make sure to REDUCE SIZE OF COMMENTS. This is very important. Every comment should be concise 1-2 liner maximum 4 lines if describes really extensive and unnatural concept.
### Important coding rules
- Do not add doc comments to the private structs and functions.
- Do not make public structs if something can be private
## Architecture
Everything that is performance critical happens in rust world, everything that is neovim specific happens in the lua code.
There are 3 main components:
- Rust binary with the global file picker state containing index of all files
- Background thread with the file system watcher that updates the index in real time
- Lua UI layer that renders the picker, handles user input, and calls the rust functions via FFI
There are 2 databases:
- Frecency database (LMDB) that tracks file access patterns for scoring
- Query history database used to track the user's previous search queries
### Key Files
- `lua/fff.lua` - Entry point, delegates to main.lua
- `lua/fff/main.lua` - Public API (find_files, search, change_directory)
- `lua/fff/core.lua` - Initialization, autocmds, global state management
- `lua/fff/picker_ui.lua` - UI rendering, layout calculation, keymaps
- `lua/fff/file_picker/preview.lua` - File preview with syntax highlighting
- `lua/fff/file_picker/image.lua` - Image preview (snacks.nvim integration)
- `lua/fff/conf.lua` - Default config
- `lua/fff/rust/init.lua` - Loads compiled Rust shared library
**Rust Side:**
- `lua/fff/rust/lib.rs` - FFI bindings, global state (FILE_PICKER, FRECENCY)
- `lua/fff/rust/file_picker.rs` - Core FilePicker struct, indexing, background watcher
- `lua/fff/rust/frecency.rs` - Frecency database (LMDB) and scoring
- `lua/fff/rust/query_tracker.rs` - Search query history tracking
- `lua/fff/rust/score.rs` - Fuzzy match scoring with frizbee integration
- `lua/fff/rust/git.rs` - Git status caching and repository detection
- `lua/fff/rust/background_watcher.rs` - File system watcher thread
### Scoring Algorithm
Located at the score.rs file
### Build System
- `Cargo.toml` - Rust dependencies and build configuration (package name: `fff_nvim`)
- `rust-toolchain.toml` - Specifies Rust nightly toolchain with required components
- `Cross.toml` - Cross-compilation settings using Zig for Linux targets
- **CI/CD Workflows**:
- `.github/workflows/rust.yml` - Rust testing, formatting, and clippy checks
- `.github/workflows/release.yaml` - Automated multi-platform builds
- `.github/workflows/stylua.yaml` - Lua code formatting validation
- `.github/workflows/nix.yml` - Nix build validation
- **Cross-compilation Support**: Uses `cross` tool with Zig backend for efficient cross-compilation
## Development Notes
### Working with Rust Code
- Prefer struct methods over functions
- If there is more than 2 impls in the file - create new file
- Smaller concise comments over giant comment blocks
- Do not add doc comments to the private functions/structs
- Be very careful around locking and better double check with the human if something is going to require potentially long lock on a mutex/rwlock
### Working with lua code
- Document the types of public functions in every module
- Use `vim.validate()` for validating user inputs in public functions
- Try to reuse as much of existing functions as possible
- When working on new features for the UI **IT IS EXTREMELY IMPORTANT** to keep the core functionality of navigating between files, selecting, and seeing the preview working as is. NEVER break anything from the core UI functionality, only add new features on top of the current UI.
- When making a large chunk of code make lua test that opens neovim at `~/dev/lightsource` and opens the picker to test the ui functionality across the actual code.
- When adding a new highlights or any new shortcuts and configurable UI options add them to the neovim config. AND IMPORTANT: update the README.md with the new configuration options.
### UI rendering
When working on the UI changeds IT IS EXTREMELY important for you to test it for both prompt_position="bottom" and prompt_position="top" as the rendering logic is different for both of them in both rust and lua world. When the prompt is positioed in the bottom everything should work the same way as the top but would be reversed in order. (though navigation is same for both)
## Top level API that can not introduce breaking changes under any circumstance
Top level rust, lua, C, and bun APIs can not be changed under any circumstance
CLAUDE.md
Symlink
+1
View File
@@ -0,0 +1 @@
AGENTS.md
Cargo.lock
Generated
+100 -53
View File
@@ -667,14 +667,15 @@ dependencies = [
[[package]]
name = "fff-notify-debouncer-full"
version = "0.9.1"
version = "0.9.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4594f0ef1ad0bfcd112bbe4b397ef881f50442bc710356b86858d555848e4d09"
checksum = "5c6f0c16164d10c082af931377766f5495440bee66a9a841bbcea1af5de8878c"
dependencies = [
"file-id",
"log",
"notify 9.0.0-rc.3",
"notify",
"notify-types",
"rustc-hash 2.1.2",
"walkdir",
]
@@ -688,6 +689,7 @@ dependencies = [
"criterion",
"ctrlc",
"dirs",
"fff-notify-debouncer-full",
"fff-query-parser",
"fff-search",
"git2",
@@ -697,11 +699,11 @@ dependencies = [
"mimalloc",
"mlua",
"neo_frizbee",
"notify 8.2.0",
"notify-debouncer-full",
"notify",
"once_cell",
"parking_lot",
"pathdiff",
"rand",
"rand 0.8.5",
"rayon",
"serde",
"smallvec",
@@ -745,11 +747,12 @@ dependencies = [
"memmap2",
"mimalloc",
"neo_frizbee",
"notify 9.0.0-rc.3",
"notify",
"once_cell",
"parking_lot",
"pathdiff",
"rand",
"proptest",
"rand 0.8.5",
"rayon",
"regex",
"regex-syntax",
@@ -781,6 +784,12 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "fnv"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
[[package]]
name = "foldhash"
version = "0.1.5"
@@ -796,15 +805,6 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "fsevent-sys"
version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76ee7a02da4d231650c7cea31349b889be2f45ddb3ef3032d2ec8185f6313fd2"
dependencies = [
"libc",
]
[[package]]
name = "futures"
version = "0.3.32"
@@ -1552,24 +1552,6 @@ dependencies = [
"minimal-lexical",
]
[[package]]
name = "notify"
version = "8.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4d3d07927151ff8575b7087f245456e549fea62edf0ec4e565a5ee50c8402bc3"
dependencies = [
"bitflags 2.11.0",
"fsevent-sys",
"inotify",
"kqueue",
"libc",
"log",
"mio",
"notify-types",
"walkdir",
"windows-sys 0.60.2",
]
[[package]]
name = "notify"
version = "9.0.0-rc.3"
@@ -1590,19 +1572,6 @@ dependencies = [
"xxhash-rust",
]
[[package]]
name = "notify-debouncer-full"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "375bd3a138be7bfeff3480e4a623df4cbfb55b79df617c055cd810ba466fa078"
dependencies = [
"file-id",
"log",
"notify 8.2.0",
"notify-types",
"walkdir",
]
[[package]]
name = "notify-types"
version = "2.1.0"
@@ -1762,7 +1731,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3c80231409c20246a13fddb31776fb942c38553c51e871f8cbd687a4cfb5843d"
dependencies = [
"phf_shared",
"rand",
"rand 0.8.5",
]
[[package]]
@@ -1870,6 +1839,29 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "proptest"
version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744"
dependencies = [
"bitflags 2.11.0",
"num-traits",
"rand 0.9.4",
"rand_chacha 0.9.0",
"rand_xorshift",
"regex-syntax",
"rusty-fork",
"tempfile",
"unarray",
]
[[package]]
name = "quick-error"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
[[package]]
name = "quote"
version = "1.0.45"
@@ -1898,8 +1890,18 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
dependencies = [
"libc",
"rand_chacha",
"rand_core",
"rand_chacha 0.3.1",
"rand_core 0.6.4",
]
[[package]]
name = "rand"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea"
dependencies = [
"rand_chacha 0.9.0",
"rand_core 0.9.5",
]
[[package]]
@@ -1909,7 +1911,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
dependencies = [
"ppv-lite86",
"rand_core",
"rand_core 0.6.4",
]
[[package]]
name = "rand_chacha"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
dependencies = [
"ppv-lite86",
"rand_core 0.9.5",
]
[[package]]
@@ -1921,6 +1933,24 @@ dependencies = [
"getrandom 0.2.17",
]
[[package]]
name = "rand_core"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
dependencies = [
"getrandom 0.3.4",
]
[[package]]
name = "rand_xorshift"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a"
dependencies = [
"rand_core 0.9.5",
]
[[package]]
name = "raw-cpuid"
version = "11.6.0"
@@ -2085,6 +2115,17 @@ version = "1.0.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
[[package]]
name = "rusty-fork"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc6bf79ff24e648f6da1f8d1f011e9cac26491b619e6b9280f2b47f1774e6ee2"
dependencies = [
"fnv",
"quick-error",
"tempfile",
]
[[package]]
name = "same-file"
version = "1.0.6"
@@ -2577,6 +2618,12 @@ dependencies = [
"tracing-log",
]
[[package]]
name = "unarray"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94"
[[package]]
name = "unicode-ident"
version = "1.0.24"
@@ -2776,7 +2823,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.61.2",
"windows-sys 0.48.0",
]
[[package]]
Cargo.toml
+9 -2
View File
@@ -7,7 +7,6 @@ members = [
"crates/fff-query-parser",
"crates/fff-grep",
]
exclude = ["crates/fff-notify-debouncer-full"]
resolver = "2"
[workspace.dependencies]
@@ -37,7 +36,7 @@ zlob = "1.3.3"
mlua = { version = "0.11.1", features = ["module", "luajit"] }
neo_frizbee = { version = "0.10.1", features = ["match_end_col"] }
notify = { version = "9.0.0-rc.3" }
notify-debouncer-full = { package = "fff-notify-debouncer-full", version = "0.9.1" }
notify-debouncer-full = { package = "fff-notify-debouncer-full", version = "0.9.3" }
once_cell = "1.20.2"
parking_lot = "0.12"
pathdiff = "0.2.1"
@@ -62,3 +61,11 @@ lto = "thin"
[profile.bench]
inherits = "release"
# For Instruments / xctrace: release-level optimization but keep debuginfo
# and symbols so sampled frames resolve to real Rust names.
[profile.prof]
inherits = "release"
debug = "full"
strip = false
lto = "thin"
Makefile
+25 -1
View File
@@ -4,7 +4,11 @@ PREFIX ?= /usr/local
LIBDIR ?= $(PREFIX)/lib
INCLUDEDIR ?= $(PREFIX)/include
.PHONY: build build-c-lib install uninstall test test-rust test-lua test-version test-bun test-node prepare-bun prepare-node set-npm-version header
# Compile-time cfg that gates the watcher + git-status fuzz stress test.
STRESS_RUSTFLAGS := --cfg stress
FFF_STRESS_DEFAULT_SEED ?= 0xDEADBEEFCAFEBABE
.PHONY: build build-c-lib install uninstall test test-rust test-lua test-version test-bun test-node prepare-bun prepare-node set-npm-version header test-stress test-stress-seeded test-stress-random
all: format test lint
@@ -85,6 +89,26 @@ test-node: prepare-node
test: test-rust test-lua test-version test-bun test-node
test-stress-seeded:
FFF_STRESS_SEED="$${FFF_STRESS_SEED:-$(FFF_STRESS_DEFAULT_SEED)}" \
RUSTFLAGS="$(STRESS_RUSTFLAGS)" \
cargo test \
-p fff-search \
--test fuzz_git_watcher_stress \
--features zlob \
-- --nocapture stress_seeded
test-stress-random:
RUSTFLAGS="$(STRESS_RUSTFLAGS)" \
cargo test \
-p fff-search \
--test fuzz_git_watcher_stress \
--features zlob \
-- --nocapture stress_random
test-stress: test-stress-seeded test-stress-random
# Update version in a package.json, including optionalDependencies.
# Usage: make set-npm-version PKG=packages/fff-bun VERSION=1.0.0-nightly.abc1234
set-npm-version:
_typos.toml
+2
View File
@@ -6,6 +6,8 @@ noice = "noice"
fo = "fo"
ba = "ba"
ue = "ue"
# file extensions that look like typos
thm = "thm"
# some typos we use for tests
comparsion = "comparsion"
modfiers = "modfiers"
crates/fff-c/src/lib.rs
+15 -20
View File
@@ -35,7 +35,7 @@ use fff::file_picker::FilePicker;
use fff::frecency::FrecencyTracker;
use fff::query_tracker::QueryTracker;
use fff::{DbHealthChecker, FFFMode, FuzzySearchOptions, PaginationArgs, QueryParser};
use fff::{SharedFrecency, SharedPicker};
use fff::{SharedFilePicker, SharedFrecency};
use ffi_types::{
FffDirItem, FffDirSearchResult, FffFileItem, FffGrepMatch, FffGrepResult, FffMixedItem,
FffMixedSearchResult, FffResult, FffScanProgress, FffScore, FffSearchResult,
@@ -46,7 +46,7 @@ use ffi_types::{
/// The caller receives this as `*mut c_void` and must pass it to every FFI call.
/// The fff_handle is freed by `fff_destroy`.
struct FffInstance {
picker: SharedPicker,
picker: SharedFilePicker,
frecency: SharedFrecency,
query_tracker: SharedQueryTracker,
}
@@ -204,7 +204,7 @@ pub unsafe extern "C" fn fff_create_instance2(
let history_path = unsafe { optional_cstr(history_db_path) }.map(|s| s.to_string());
// Create shared state that background threads will write into.
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
let query_tracker = SharedQueryTracker::default();
@@ -744,18 +744,10 @@ pub unsafe extern "C" fn fff_scan_files(fff_handle: *mut c_void) -> *mut FffResu
Err(e) => return e,
};
let mut guard = match inst.picker.write() {
Ok(g) => g,
Err(e) => return FffResult::err(&format!("Failed to acquire file picker lock: {}", e)),
};
let picker = match guard.as_mut() {
Some(p) => p,
None => return FffResult::err("File picker not initialized"),
};
match picker.trigger_rescan(&inst.frecency) {
Ok(_) => FffResult::ok_empty(),
// Async: rescan runs on a BG thread, caller returns immediately.
// Use `fff_is_scanning` / `fff_wait_for_scan` to observe progress.
match inst.picker.trigger_full_rescan_async(&inst.frecency) {
Ok(()) => FffResult::ok_empty(),
Err(e) => FffResult::err(&format!("Failed to trigger rescan: {}", e)),
}
}
@@ -904,14 +896,17 @@ pub unsafe extern "C" fn fff_restart_index(
};
let (warmup_caches, content_indexing, watch, mode) = if let Some(mut picker) = guard.take() {
let warmup = picker.need_enable_mmap_cache();
let ci = picker.need_enable_content_indexing();
let w = picker.need_watch();
let warmup = picker.has_mmap_cache();
let enable_content_indexing = picker.has_content_indexing();
let watch = picker.has_watcher();
let mode = picker.mode();
picker.stop_background_monitor();
(warmup, ci, w, mode)
(warmup, enable_content_indexing, watch, mode)
} else {
(false, false, true, FFFMode::default())
// this is error state anyway
(false, true, true, FFFMode::default())
};
drop(guard);
crates/fff-core/Cargo.toml
+1
View File
@@ -79,5 +79,6 @@ dunce = { workspace = true }
[dev-dependencies]
criterion = { version = "0.5", features = ["html_reports"] }
proptest = { version = "1", default-features = false, features = ["std", "fork"] }
rand = { version = "0.8", features = ["small_rng"] }
tempfile = "3.8"
crates/fff-core/benches/bigram_bench.rs
+37 -2
View File
@@ -90,7 +90,8 @@ fn bench_bigram_build(c: &mut Criterion) {
let file_counts = [10_000, 100_000];
for &file_count in &file_counts {
// Pre-generate content so we only measure index building
// Pre-generate content so we only measure index building.
// Short content (~85 bytes/file) exercises the scalar fast path.
let contents: Vec<String> = (0..file_count)
.map(|i| {
format!(
@@ -100,7 +101,7 @@ fn bench_bigram_build(c: &mut Criterion) {
.collect();
group.bench_with_input(
BenchmarkId::new("build_and_compress", file_count),
BenchmarkId::new("short_content", file_count),
&file_count,
|b, &fc| {
b.iter(|| {
@@ -114,6 +115,40 @@ fn bench_bigram_build(c: &mut Criterion) {
});
},
);
// Long content (~4 KB/file) exercises the SIMD pre-pass path.
// Build a realistic-looking source-like blob by repeating snippets.
let long_contents: Vec<String> = (0..file_count)
.map(|i| {
let mut s = String::with_capacity(4096);
for j in 0..50 {
s.push_str(&format!(
"pub fn handler_{i}_{j}(ctx: &Context) -> Result<Response, Error> {{\n"
));
s.push_str(" let parsed = ctx.parse()?;\n");
s.push_str(" let validated = parsed.validate()?;\n");
s.push_str(&format!(" ctx.respond(validated, {}).await\n", j));
s.push_str("}\n\n");
}
s
})
.collect();
group.bench_with_input(
BenchmarkId::new("long_content", file_count),
&file_count,
|b, &fc| {
b.iter(|| {
let builder = BigramIndexBuilder::new(fc);
let skip_builder = BigramIndexBuilder::new(fc);
for (i, content) in long_contents.iter().enumerate() {
builder.add_file_content(&skip_builder, i, content.as_bytes());
}
let index = builder.compress(None);
black_box(index.columns_used())
});
},
);
}
group.finish();
crates/fff-core/build.rs
+8
View File
@@ -1,4 +1,12 @@
fn main() {
// Opt-in cfg for the long-running randomized stress tests
// used by tests/fuzz_git_watcher_stress.rs
println!("cargo::rustc-check-cfg=cfg(stress)");
// When the `zlob` feature is enabled (Zig-compiled C library):
// On Windows MSVC, explicitly link the C runtime libraries.
// Zig-compiled static libraries don't emit /DEFAULTLIB directives for the
// MSVC CRT, so symbols like strcmp, memcpy etc. would be unresolved.
if std::env::var("CARGO_FEATURE_ZLOB").is_ok() {
if !zig_available() {
panic!(
crates/fff-core/src/background_watcher.rs
+355 -185
View File
@@ -1,15 +1,15 @@
use crate::error::Error;
use crate::file_picker::{FFFMode, FilePicker};
use crate::git::GitStatusCache;
use crate::shared::{SharedFrecency, SharedPicker};
use crate::shared::{SharedFilePicker, SharedFrecency};
use crate::sort_buffer::sort_with_buffer;
use git2::Repository;
use notify::event::{AccessKind, AccessMode};
use notify::{Config, EventKind, EventKindMask, RecursiveMode};
use notify_debouncer_full::{DebounceEventResult, DebouncedEvent, NoCache, new_debouncer_opt};
use parking_lot::Mutex;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc;
use std::time::Duration;
use tracing::{Level, debug, error, info, warn};
@@ -18,24 +18,13 @@ type Debouncer = notify_debouncer_full::Debouncer<notify::RecommendedWatcher, No
/// Owns the file-system watcher and guarantees that all background threads
/// are fully joined before `stop()` / `Drop` returns.
///
/// Architecture:
/// - The debouncer (and its internal watcher) live inside an **owner thread**
/// that we spawn and hold the `JoinHandle` for.
/// - `stop()` sets a flag, unparks the owner thread, and **joins** it.
/// - Inside the owner thread, `Debouncer::stop()` is called which joins the
/// debouncer's event-processing thread.
/// - On Windows an additional short sleep is added after `Debouncer::stop()`
/// because `notify`'s `ReadDirectoryChangesWatcher` discards its thread
/// `JoinHandle`, so we cannot join it directly. The watcher's `Drop` does
/// signal the thread via semaphore so it exits almost immediately, but we
/// need to give the OS a moment to reclaim it.
pub struct BackgroundWatcher {
stop_signal: Arc<AtomicBool>,
debouncer: Arc<Mutex<Option<Debouncer>>>,
watch_tx: Option<mpsc::Sender<PathBuf>>,
owner_thread: Option<std::thread::JoinHandle<()>>,
}
const DEBOUNCE_TIMEOUT: Duration = Duration::from_millis(250);
const DEBOUNCE_TIMEOUT: Duration = Duration::from_millis(50);
const MAX_PATHS_THRESHOLD: usize = 1024;
/// On macOS, each `watch()` call creates a separate FSEventStream. When the
/// number of directories exceeds this threshold we fall back to a single
@@ -44,15 +33,15 @@ const MAX_MACOS_NONRECURSIVE_WATCHES: usize = 4096;
/// Minimum seconds between frecency tracks of the same file in AI mode.
/// Prevents score inflation from rapid burst edits by AI agents.
const AI_MODE_COOLDOWN_SECS: u64 = 5 * 60;
const MAX_OVERFLOW_FILES: usize = 1024;
impl BackgroundWatcher {
pub fn new(
base_path: PathBuf,
git_workdir: Option<PathBuf>,
shared_picker: SharedPicker,
shared_picker: SharedFilePicker,
shared_frecency: SharedFrecency,
mode: FFFMode,
watch_dirs: Vec<PathBuf>,
) -> Result<Self, Error> {
info!(
"Initializing background watcher for path: {}, mode: {:?}",
@@ -60,11 +49,41 @@ impl BackgroundWatcher {
mode,
);
let (watch_tx, watch_rx) = mpsc::channel::<PathBuf>();
// Refuse to watch the filesystem root or the user's home directory.
// These are prone to high-volume event churn (editor temp files,
// browser caches, log rotations) which inflates the overflow arena
// and, on macOS, can exhaust the per-process FSEvents stream limit.
if base_path.parent().is_none()
|| Some(base_path.as_os_str()) == dirs::home_dir().as_ref().map(|p| p.as_os_str())
{
return Err(Error::FilesystemRoot(base_path));
}
// Clone shared state for the owner thread (needed for injecting
// files that existed before a watch was registered on their directory).
let owner_picker = shared_picker.clone();
// macOS: always use a single recursive FSEvent stream.
//
// Per-dir NonRecursive watches create one FSEvent stream per dir.
// The per-process FSEvent cap is lower than expected in practice
// (4096 per process, but FFF usually is running within code editors),
// and each failed `watch()` after the cap blocks ~40 ms on kernel retry.
// Yes we pay for filtering events on handler phase but it is usable
//
// macOS and Windows use a single recursive watch. FSEvents and
// ReadDirectoryChangesW both support true kernel-level recursion
// on one handle — per-dir NonRecursive watches burn streams/handles
// for no benefit and, on Windows, have been observed to silently
// drop Modify events for nested paths.
//
// Linux keeps the per-dir NonRecursive strategy: inotify has no
// kernel-level recursion, so Recursive here would still register
// one watch per subdir but without the ignored-dir filtering we
// get by iterating `picker.for_each_dir` ourselves.
let use_recursive = cfg!(any(target_os = "macos", target_os = "windows"));
let (watch_tx, watch_rx) = mpsc::channel::<PathBuf>();
let watch_tx_for_debouncer = watch_tx.clone();
let owner_weak_picker = shared_picker.weaken();
let owner_frecency = shared_frecency.clone();
let owner_git_workdir = git_workdir.clone();
let debouncer = Self::create_debouncer(
@@ -73,57 +92,73 @@ impl BackgroundWatcher {
shared_picker,
shared_frecency,
mode,
watch_dirs,
watch_tx,
use_recursive,
watch_tx_for_debouncer,
)?;
info!("Background file watcher initialized successfully");
let stop_signal = Arc::new(AtomicBool::new(false));
let stop_clone = Arc::clone(&stop_signal);
// debouncer is shared with the owner thread, once it's dropped the thread is closed
let debouncer = Arc::new(Mutex::new(Some(debouncer)));
// Only the Linux per-dir-watch branch needs this clone; on other
// platforms the owner thread never touches the debouncer.
#[cfg(target_os = "linux")]
let owner_debouncer = Arc::clone(&debouncer);
// The owner thread keeps the debouncer alive and ensures proper
// cleanup: `Debouncer::stop()` joins its internal thread, then the
// watcher `Drop` signals its I/O thread to exit.
let owner_thread = std::thread::Builder::new()
.name("fff-watcher-owner".into())
.name("fff-watcher-own".into())
.spawn(move || {
let mut debouncer = debouncer;
while !stop_clone.load(Ordering::Acquire) {
// Process pending watch requests from the event handler
// (new directories that need to be watched).
while let Ok(dir) = watch_rx.try_recv() {
match debouncer.watch(dir.as_path(), RecursiveMode::NonRecursive) {
Ok(()) => {
debug!("Added watch for new directory: {}", dir.display());
}
Err(e) => {
warn!("Failed to watch new directory {}: {}", dir.display(), e);
}
}
while let Ok(dir) = watch_rx.recv() {
// if the picker is dropped we do need to exit the loop
let Some(strong_picker) = owner_weak_picker.upgrade() else {
break;
};
// Files created before the watch was registered don't
// generate events. Do a flat (non-recursive) read_dir
// to inject any files that already exist. Subdirectories
// are not descended — they get their own watches via
// future Create events from this directory's watch.
inject_existing_files(&dir, &owner_picker, &owner_git_workdir);
// Only inotify (Linux) has no kernel-level recursion, so
// it's the only platform that needs a per-subdir watch to
// be registered at runtime. macOS FSEvents and Windows
// ReadDirectoryChangesW are already watching recursively
// from the base path (see `create_debouncer`), and
// registering a second overlapping stream there produces
// duplicate/out-of-order events.
#[cfg(target_os = "linux")]
{
// Register the new directory with the debouncer, then
// drop the mutex BEFORE doing picker-side work — see
// the comment on `BackgroundWatcher::stop` for the
// lock-ordering rationale.
let mut guard = owner_debouncer.lock();
let Some(debouncer) = guard.as_mut() else {
break;
};
if let Err(e) = debouncer.watch(&dir, RecursiveMode::NonRecursive) {
warn!(
?e,
dir = %dir.display(),
"Failed to init watcher for new directory"
);
}
}
std::thread::park_timeout(Duration::from_secs(1));
track_files_from_new_directories(
&dir,
&strong_picker,
&owner_frecency,
&owner_git_workdir,
);
// Transient strong ref drops here, back
// to weak-only before the next `recv()`.
}
// Debouncer::stop() joins the debouncer's event thread, then
// drops the watcher (whose Drop signals the I/O thread).
debouncer.stop();
// On Windows the notify crate discards the ReadDirectoryChangesW
// thread's JoinHandle — we cannot join it. Its Drop signals the
// thread via semaphore so it exits almost immediately; give the
// OS a moment to fully reclaim it.
#[cfg(windows)]
std::thread::sleep(Duration::from_millis(250));
tracing::info!("Background watcher is stopped");
})
.expect("failed to spawn fff-watcher-owner thread");
Ok(Self {
stop_signal,
debouncer,
watch_tx: Some(watch_tx),
owner_thread: Some(owner_thread),
})
}
@@ -131,10 +166,10 @@ impl BackgroundWatcher {
fn create_debouncer(
base_path: PathBuf,
git_workdir: Option<PathBuf>,
shared_picker: SharedPicker,
shared_picker: SharedFilePicker,
shared_frecency: SharedFrecency,
mode: FFFMode,
watch_dirs: Vec<PathBuf>,
use_recursive: bool,
watch_tx: mpsc::Sender<PathBuf>,
) -> Result<Debouncer, Error> {
let config = Config::default()
@@ -146,33 +181,44 @@ impl BackgroundWatcher {
// our own grep calls and preview window rendering
.with_event_kinds(EventKindMask::CORE);
// Decide the watching strategy up-front so the event handler closure
// knows whether it needs to request dynamic directory watches.
let use_recursive =
cfg!(target_os = "macos") && watch_dirs.len() > MAX_MACOS_NONRECURSIVE_WATCHES;
// `use_recursive` was decided by the caller from a cheap size hint,
// so the event-handler closure can capture it directly.
//
// The closure lives on the debouncer's internal event thread
// for as long as the debouncer exists — i.e. the full
// lifetime of `BackgroundWatcher`. Capturing a strong
// `SharedFilePicker` here would re-introduce the Arc cycle
// we just broke with `owner_picker`'s `downgrade()` above.
// Capture a weak handle instead and upgrade per-batch.
let git_workdir_for_handler = git_workdir.clone();
let shared_picker_for_watching = shared_picker.clone();
let event_picker = shared_picker.weaken();
let mut debouncer = new_debouncer_opt(
DEBOUNCE_TIMEOUT,
Some(DEBOUNCE_TIMEOUT / 2), // tick rate for the event span
{
move |result: DebounceEventResult| match result {
Ok(events) => {
// Upgrade just long enough to drive one
// debounced batch. Failure means every
// external `SharedFilePicker` has already
// dropped and teardown is already underway.
let Some(strong_picker) = event_picker.upgrade() else {
return;
};
let new_dirs = handle_debounced_events(
events,
&git_workdir_for_handler,
&shared_picker,
&strong_picker,
&shared_frecency,
mode,
);
// In NonRecursive mode, register watches for newly
// discovered directories so future file events in them
// are captured. In Recursive mode the single stream
// already covers new subdirectories.
if !use_recursive {
for dir in new_dirs {
let _ = watch_tx.send(dir);
// every new directory creates had to be reflected in the picker state
for dir in new_dirs {
if let Err(e) = watch_tx.send(dir) {
warn!(?e, "Failed to send directory update error");
}
}
}
@@ -215,28 +261,70 @@ impl BackgroundWatcher {
debouncer.watch(base_path.as_path(), RecursiveMode::Recursive)?;
info!(
"File watcher initialized with single recursive watch on {} \
({} directories exceeded threshold of {})",
(exceeded threshold of {})",
base_path.display(),
watch_dirs.len(),
MAX_MACOS_NONRECURSIVE_WATCHES,
);
} else {
debouncer.watch(base_path.as_path(), RecursiveMode::NonRecursive)?;
for dir in &watch_dirs {
match debouncer.watch(dir.as_path(), RecursiveMode::NonRecursive) {
Ok(()) => {}
Err(e) => {
// Non-fatal: directory may have been removed between discovery and watch
warn!("Failed to watch directory {}: {}", dir.display(), e);
// Stream watch-dir registration directly under the picker
// read lock. Only Linux (inotify) reaches this branch —
// macOS always takes the recursive path above. `inotify`'s
// `inotify_add_watch()` is fast-fail: on ENOSPC it returns
// immediately, no kernel retry loop, so holding the read
// lock across the stream is O(ms) even for large repos.
//
// Abort the loop after a run of failures. Once ENOSPC hits,
// further calls won't succeed until the user raises
// `fs.inotify.max_user_watches`, so there's no value in
// continuing.
const MAX_CONSECUTIVE_WATCH_FAILURES: usize = 16;
let mut watched = 0usize;
let mut consecutive_failures = 0usize;
let mut aborted_early = false;
if let Some(guard) = shared_picker_for_watching.read().ok()
&& let Some(picker) = guard.as_ref()
{
use std::ops::ControlFlow;
picker.for_each_dir(|dir| {
match debouncer.watch(dir, RecursiveMode::NonRecursive) {
Ok(()) => {
watched += 1;
consecutive_failures = 0;
ControlFlow::Continue(())
}
Err(e) => {
consecutive_failures += 1;
if consecutive_failures <= 4 {
warn!("Failed to watch directory {}: {}", dir.display(), e);
}
if consecutive_failures >= MAX_CONSECUTIVE_WATCH_FAILURES {
warn!(
consecutive_failures,
watched,
"Aborting NonRecursive watch loop — per-process \
watch cap exhausted, further dirs would just burn \
kernel time for no coverage"
);
aborted_early = true;
ControlFlow::Break(())
} else {
ControlFlow::Continue(())
}
}
}
}
});
}
info!(
"File watcher initialized for {} directories (NonRecursive) under {}",
watch_dirs.len(),
base_path.display()
"File watcher initialized for {} directories (NonRecursive) under {} (aborted_early={})",
watched,
base_path.display(),
aborted_early,
);
}
@@ -252,17 +340,50 @@ impl BackgroundWatcher {
Ok(debouncer)
}
/// Signal the watcher to shut down without blocking on its worker
/// threads. Safe to call from any context, including while holding
/// the [`SharedFilePicker`] write lock.
///
/// Both the debouncer's internal event loop and our owner thread
/// may call `SharedFilePicker::write()` inside their handlers. A
/// blocking join here would deadlock against a caller that already
/// holds that lock (e.g. `stop_background_monitor` under a
/// `shared_picker.write()` guard). Instead we:
///
/// * drop the `watch_tx` Sender — the owner thread's
/// `watch_rx.recv()` returns `Err` and the thread exits at
/// its next `recv`.
/// * call `debouncer.stop_nonblocking()` — signals the debouncer
/// event loop to exit on its next tick and drops the watcher,
/// closing the FSEvent / inotify / ReadDirectoryChangesW stream.
/// * detach both `JoinHandle`s.
///
/// In-flight handler invocations finish on their own (at most one
/// more batch) once the caller releases any locks they hold.
pub fn stop(&mut self) {
self.stop_signal.store(true, Ordering::Release);
if let Some(handle) = self.owner_thread.take() {
handle.thread().unpark();
if let Err(e) = handle.join() {
error!("Watcher owner thread panicked: {:?}", e);
}
self.watch_tx.take();
if let Some(debouncer) = self.debouncer.lock().take() {
debouncer.stop_nonblocking();
}
info!("Background file watcher stopped successfully");
self.owner_thread.take();
info!("Background file watcher stop signaled");
}
/// Queue a non-recursive watch registration on `dir`.
///
/// The owner thread is always blocked on `watch_rx.recv()`, so
/// the `send()` here wakes it immediately via the channel's
/// condvar — no external unpark needed.
///
/// Returns `false` once `stop()` has dropped our `Sender` — any
/// further request is silently discarded.
pub(crate) fn request_watch_dir(&self, dir: PathBuf) -> bool {
match self.watch_tx.as_ref() {
Some(tx) => tx.send(dir).is_ok(),
None => false,
}
}
}
@@ -276,7 +397,7 @@ impl Drop for BackgroundWatcher {
fn handle_debounced_events(
events: Vec<DebouncedEvent>,
git_workdir: &Option<PathBuf>,
shared_picker: &SharedPicker,
shared_picker: &SharedFilePicker,
shared_frecency: &SharedFrecency,
mode: FFFMode,
) -> Vec<PathBuf> {
@@ -285,6 +406,7 @@ fn handle_debounced_events(
let mut need_full_rescan = false;
let mut need_full_git_rescan = false;
let mut paths_to_remove = Vec::new();
let mut dirs_to_remove: Vec<PathBuf> = Vec::new();
let mut paths_to_add_or_modify = Vec::new();
let mut new_dirs_to_watch = Vec::new();
let mut affected_paths_count = 0usize;
@@ -346,8 +468,20 @@ fn handle_debounced_events(
// - Remove events are not always emitted (macOS often sends
// Modify(Name(Any)) instead of Remove).
let is_removal = matches!(debounced_event.event.kind, EventKind::Remove(_));
// Directory-level remove: macOS FSEvents delivers a single
// `Remove(Folder)` event for a whole directory tree (e.g.
// after `git reset --hard` wipes a dir full of staged-but-
// uncommitted files). Individual per-file Remove events for
// the children do *not* arrive. Treat the folder removal as
// "evict every indexed descendant".
let is_folder_removal = matches!(
debounced_event.event.kind,
EventKind::Remove(notify::event::RemoveKind::Folder)
);
if is_removal || !path.exists() {
if is_folder_removal {
dirs_to_remove.push(path.to_path_buf());
} else if is_removal || !path.exists() {
paths_to_remove.push(path.as_path());
} else if path.is_dir() {
// New directory — collect it so the caller can register a
@@ -382,7 +516,9 @@ fn handle_debounced_events(
if need_full_rescan {
info!(?affected_paths_count, "Triggering full rescan");
trigger_full_rescan(shared_picker, shared_frecency);
if let Err(e) = shared_picker.trigger_full_rescan_async(shared_frecency) {
error!("Failed to trigger full rescan: {:?}", e);
}
return Vec::new();
}
@@ -393,59 +529,90 @@ fn handle_debounced_events(
paths_to_add_or_modify.dedup_by(|a, b| a.as_os_str().eq(b.as_os_str()));
info!(
"Event processing summary: {} to remove, {} to add/modify, {} new dirs",
"Event processing summary: {} to remove, {} dirs to remove, {} to add/modify, {} new dirs",
paths_to_remove.len(),
dirs_to_remove.len(),
paths_to_add_or_modify.len(),
new_dirs_to_watch.len()
);
// Apply file index updates (add/remove) unconditionally — these must
// happen even when there is no git repository.
let files_to_update_git_status =
if !paths_to_remove.is_empty() || !paths_to_add_or_modify.is_empty() {
debug!(
"Applying file index changes: {} to remove, {} to add/modify",
paths_to_remove.len(),
paths_to_add_or_modify.len(),
let (files_to_update_git_status, overflow_count) = if !paths_to_remove.is_empty()
|| !dirs_to_remove.is_empty()
|| !paths_to_add_or_modify.is_empty()
{
debug!(
"Applying file index changes: {} to remove, {} dirs to remove, {} to add/modify",
paths_to_remove.len(),
dirs_to_remove.len(),
paths_to_add_or_modify.len(),
);
let apply_changes = |picker: &mut FilePicker| -> (Vec<PathBuf>, usize) {
// Remove whole directories first so any subsequent single-file
// remove event for a path that lived under them becomes a cheap
// no-op rather than a failed lookup.
for dir in &dirs_to_remove {
let count = picker.remove_all_files_in_dir(dir);
debug!("remove_all_files_in_dir({:?}) -> {} files", dir, count);
}
for path in &paths_to_remove {
let removed = picker.remove_file_by_path(path);
debug!("remove_file_by_path({:?}) -> {}", path, removed);
}
let mut files_to_update = Vec::with_capacity(paths_to_add_or_modify.len());
for path in &paths_to_add_or_modify {
let added = picker.on_create_or_modify(path).is_some();
if added {
debug!("on_create_or_modify({:?}) -> Some", path);
files_to_update.push(path.to_path_buf());
} else {
error!("on_create_or_modify({:?}) -> None (file not added!)", path);
}
}
let overflow_count = picker.get_overflow_files().len();
info!(
"apply_changes complete: {} files to update git status, overflow={}",
files_to_update.len(),
overflow_count,
);
let apply_changes = |picker: &mut FilePicker| -> Vec<PathBuf> {
for path in &paths_to_remove {
let removed = picker.remove_file_by_path(path);
debug!("remove_file_by_path({:?}) -> {}", path, removed);
}
let mut files_to_update = Vec::with_capacity(paths_to_add_or_modify.len());
for path in &paths_to_add_or_modify {
let added = picker.on_create_or_modify(path).is_some();
if added {
debug!("on_create_or_modify({:?}) -> Some", path);
files_to_update.push(path.to_path_buf());
} else {
error!("on_create_or_modify({:?}) -> None (file not added!)", path);
}
}
info!(
"apply_changes complete: {} files to update git status",
files_to_update.len()
);
files_to_update
};
let Ok(mut guard) = shared_picker.write() else {
error!("Failed to acquire file picker write lock");
return new_dirs_to_watch;
};
let Some(ref mut picker) = *guard else {
error!("File picker not initialized");
return new_dirs_to_watch;
};
apply_changes(picker)
} else {
debug!("No file index changes to apply");
Vec::new()
(files_to_update, overflow_count)
};
let Ok(mut guard) = shared_picker.write() else {
error!("Failed to acquire file picker write lock");
return new_dirs_to_watch;
};
let Some(ref mut picker) = *guard else {
error!("File picker not initialized");
return new_dirs_to_watch;
};
apply_changes(picker)
} else {
debug!("No file index changes to apply");
(Vec::new(), 0)
};
// The overflow arena grows monotonically as new files are created — a
// file's chunks are added on creation but never reclaimed on removal.
// On directories with high churn (e.g. `$HOME` with editor temp files,
// browser caches) this inflates RSS unboundedly. Once overflow exceeds
// the threshold, fall back to a full rescan: that replaces `sync_data`
// and drops the builder arena, which is the only path that reclaims it.
if overflow_count > MAX_OVERFLOW_FILES {
warn!(
?overflow_count,
"Overflow count exceeded the threshold, triggering full rescan.",
);
if let Err(e) = shared_picker.trigger_full_rescan_async(shared_frecency) {
error!("Failed to trigger full rescan: {:?}", e);
}
return new_dirs_to_watch;
}
// AI mode: auto-track frecency for all modified/created files.
// Uses a 5-minute cooldown per file to prevent score inflation from rapid
// burst edits (AI agents often edit the same file many times in minutes).
@@ -499,11 +666,19 @@ fn handle_debounced_events(
if need_full_git_rescan {
info!("Triggering full git rescan");
let result = shared_picker.refresh_git_status(shared_frecency);
if let Err(e) = result {
if let Err(e) = shared_picker.refresh_git_status(shared_frecency) {
error!("Failed to refresh git status: {:?}", e);
}
return new_dirs_to_watch;
// IMPORTANT: do NOT return here. When a batch contains both
// `.git/index` events (e.g. from `git add`) AND worktree-file
// Modify events (e.g. a subsequent edit to the same file),
// `refresh_git_status` might run while libgit2 sees an
// intermediate state — lock-wait mitigates this but can't fully
// eliminate it, and refresh doesn't always observe the final
// worktree contents if the edit event landed just before the
// batch flushed. Re-running the per-path query for explicitly
// changed files overrides any stale bits from refresh with an
// authoritative per-file status read.
}
if !files_to_update_git_status.is_empty() {
@@ -536,40 +711,14 @@ fn handle_debounced_events(
new_dirs_to_watch
}
fn trigger_full_rescan(shared_picker: &SharedPicker, shared_frecency: &SharedFrecency) {
info!("Triggering full filesystem rescan");
// Note: no need to clear mmaps — they are backed by the kernel page cache
// and automatically reflect file changes. Old FileItems (and their mmaps)
// are dropped when the picker rebuilds its file list.
let Ok(mut guard) = shared_picker.write() else {
error!("Failed to acquire file picker write lock for full rescan");
return;
};
let Some(ref mut picker) = *guard else {
error!("File picker not initialized, cannot trigger rescan");
return;
};
if let Err(e) = picker.trigger_rescan(shared_frecency) {
error!("Failed to trigger full rescan: {:?}", e);
return;
}
info!("Full filesystem rescan completed successfully");
// Spawn background warmup + bigram rebuild (mirrors the initial scan's
// post-scan phase). The write lock is still held here but the spawned
// thread re-acquires it later — safe because the guard drops at function end.
// NOTE: must NOT call shared_picker.need_complex_rebuild() here — that would
// try to read-lock the same RwLock we already hold as write, causing a deadlock.
if picker.need_enable_mmap_cache() || picker.need_enable_content_indexing() {
picker.spawn_post_rescan_rebuild(shared_picker.clone());
}
}
/// After registering a watch on a newly created directory, list its
/// immediate children and add any files to the picker.
fn inject_existing_files(dir: &Path, shared_picker: &SharedPicker, git_workdir: &Option<PathBuf>) {
fn track_files_from_new_directories(
dir: &Path,
shared_picker: &SharedFilePicker,
shared_frecency: &SharedFrecency,
git_workdir: &Option<PathBuf>,
) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
@@ -590,15 +739,36 @@ fn inject_existing_files(dir: &Path, shared_picker: &SharedPicker, git_workdir:
return;
}
let Ok(mut guard) = shared_picker.write() else {
return;
};
let Some(ref mut picker) = *guard else {
return;
};
// brief read lock
{
let Ok(mut guard) = shared_picker.write() else {
return;
};
for path in &files_to_add {
picker.on_create_or_modify(path);
let Some(ref mut picker) = *guard else {
return;
};
for path in &files_to_add {
picker.on_create_or_modify(path);
}
}
if let Some(repo) = repo.as_ref() {
let status = match GitStatusCache::git_status_for_paths(repo, &files_to_add) {
Ok(status) => status,
Err(e) => {
tracing::error!(?e, "inject_existing_files: git status query failed");
return;
}
};
if let Ok(mut guard) = shared_picker.write()
&& let Some(ref mut picker) = *guard
&& let Err(e) = picker.update_git_statuses(status, shared_frecency)
{
error!("inject_existing_files: failed to update git statuses: {e:?}");
}
}
debug!(
@@ -638,7 +808,7 @@ fn is_path_ignored(path: &Path, repo: &Option<Repository>) -> bool {
}
#[inline]
fn is_git_file(path: &Path) -> bool {
pub(crate) fn is_git_file(path: &Path) -> bool {
path.components()
.any(|component| component.as_os_str() == ".git")
}
crates/fff-core/src/bigram_filter.rs
+710 -96
View File
@@ -1,6 +1,9 @@
use std::sync::atomic::{AtomicU16, AtomicU64, AtomicUsize, Ordering};
use ahash::AHashMap;
use rayon::iter::{IndexedParallelIterator, ParallelIterator};
use rayon::slice::ParallelSlice;
use std::cell::UnsafeCell;
use std::sync::OnceLock;
use std::sync::atomic::{AtomicU16, AtomicUsize, Ordering};
/// Maximum number of distinct bigrams tracked in the inverted index.
/// 95 printable ASCII chars (32..=126) after lowercasing → ~70 distinct → 4900 possible.
@@ -17,24 +20,27 @@ pub struct BigramIndexBuilder {
// we use lookup as atomics only in the builder because it is filled by the rayon threads
// the actual index uses pure u16 for the allocations
lookup: Vec<AtomicU16>,
/// Per-column bitset data, lazily allocated via OnceLock.
col_data: Vec<AtomicU64>,
/// Flat bitset data, materialised on first use.
col_data: OnceLock<UnsafeCell<Box<[u64]>>>,
next_column: AtomicU16,
words: usize,
file_count: usize,
populated: AtomicUsize,
}
// SAFETY: `col_data`'s interior mutability is coordinated via disjoint
// `word_idx` ranges (word-aligned file partitioning in the driver), so
// concurrent access is safe despite the `UnsafeCell`. See builder doc.
unsafe impl Sync for BigramIndexBuilder {}
impl BigramIndexBuilder {
pub fn new(file_count: usize) -> Self {
let words = file_count.div_ceil(64);
let mut lookup = Vec::with_capacity(65536);
lookup.resize_with(65536, || AtomicU16::new(NO_COLUMN));
let mut col_data = Vec::with_capacity(MAX_BIGRAM_COLUMNS * words);
col_data.resize_with(MAX_BIGRAM_COLUMNS * words, || AtomicU64::new(0));
Self {
lookup,
col_data,
col_data: OnceLock::new(),
next_column: AtomicU16::new(0),
words,
file_count,
@@ -42,6 +48,23 @@ impl BigramIndexBuilder {
}
}
/// Lazily materialise the full `MAX_BIGRAM_COLUMNS * words` bitset
/// on first access.
#[inline(always)]
fn col_data_cell(&self) -> &UnsafeCell<Box<[u64]>> {
self.col_data.get_or_init(|| {
let total = MAX_BIGRAM_COLUMNS * self.words;
UnsafeCell::new(vec![0u64; total].into_boxed_slice())
})
}
/// Raw pointer to the start of the bitset slab. Used for in-place
/// `|=` writes under the partitioning invariant.
#[inline(always)]
fn col_data_ptr(&self) -> *mut u64 {
unsafe { (*self.col_data_cell().get()).as_mut_ptr() }
}
#[inline]
fn get_or_alloc_column(&self, key: u16) -> u16 {
let current = self.lookup[key as usize].load(Ordering::Relaxed);
@@ -64,13 +87,36 @@ impl BigramIndexBuilder {
}
}
#[inline]
fn column_bitset(&self, col: u16) -> &[AtomicU64] {
let start = col as usize * self.words;
&self.col_data[start..start + self.words]
/// SAFETY: caller must not access the same `word_idx` slot from
/// another thread concurrently. Partitioning in
/// `file_picker::build_bigram_index` enforces this.
#[inline(always)]
unsafe fn column_word_ptr(&self, col: u16, word_idx: usize) -> *mut u64 {
unsafe {
self.col_data_ptr()
.add(col as usize * self.words + word_idx)
}
}
pub(crate) fn add_file_content(&self, skip_builder: &Self, file_idx: usize, content: &[u8]) {
/// Test/bench accessor for a column's raw bitset words. Assumes the
/// caller has joined all writers (no concurrent mutation).
#[cfg(test)]
fn column_bitset(&self, col: u16) -> &[u64] {
let start = col as usize * self.words;
let slab = unsafe { &*self.col_data_cell().get() };
&slab[start..start + self.words]
}
// `pub` (via `#[doc(hidden)]`) only so the criterion bench can drive
// `add_file_content` directly. External consumers should use
// `build_bigram_index` instead.
///
/// SAFETY: concurrent callers must partition `file_idx` by
/// word-aligned ranges so that `file_idx / 64` never collides across
/// threads. The `file_picker::build_bigram_index` driver enforces
/// this via `par_chunks` with a word-aligned chunk size.
#[doc(hidden)]
pub fn add_file_content(&self, skip_builder: &Self, file_idx: usize, content: &[u8]) {
if content.len() < 2 {
return;
}
@@ -79,68 +125,75 @@ impl BigramIndexBuilder {
let word_idx = file_idx / 64;
let bit_mask = 1u64 << (file_idx % 64);
// Stack-local dedup bitsets: 1024 × u64 = 8 KB each, covers all 65536 bigrams with margin
// have to fit in L1 cache
// Stack-local dedup bitsets: 1024 × u64 = 8 KB each, covers all 65536
// bigram keys with margin. Has to fit in L1 cache.
let mut seen_consec = [0u64; 1024];
let mut seen_skip = [0u64; 1024];
// Normalise each byte as we stream and carry a 2-byte history
// across iterations so each input byte is normalised exactly once
// even though it participates in up to three bigrams (as `cur`,
// then `prev`, then `skip_prev`). Benchmarked against a NEON
// pre-pass variant — the pre-pass needs a heap scratch per call,
// which kills throughput unless content is gigantic. Inline
// normalisation is the faster choice for realistic file sizes.
let bytes = content;
let len = bytes.len();
// First consecutive pair (no skip bigram possible yet).
let (a, b) = (bytes[0], bytes[1]);
if (32..=126).contains(&a) && (32..=126).contains(&b) {
let key = (a.to_ascii_lowercase() as u16) << 8 | b.to_ascii_lowercase() as u16;
let w = key as usize >> 6;
let bit = 1u64 << (key as usize & 63);
seen_consec[w] |= bit;
let col = self.get_or_alloc_column(key);
if col != NO_COLUMN {
self.column_bitset(col)[word_idx].fetch_or(bit_mask, Ordering::Relaxed);
}
let mut n0 = normalize_byte_scalar(bytes[0]);
let mut n1 = normalize_byte_scalar(bytes[1]);
if n0 != u16::MAX && n1 != u16::MAX {
let key = (n0 << 8) | n1;
self.record_bigram(&mut seen_consec, key, word_idx, bit_mask);
}
// Main loop: consecutive (i-1, i) and skip-1 (i-2, i)
for i in 2..len {
let cur = bytes[i];
// Consecutive bigram: (bytes[i-1], bytes[i])
let prev = bytes[i - 1];
if (32..=126).contains(&prev) && (32..=126).contains(&cur) {
let key = (prev.to_ascii_lowercase() as u16) << 8 | cur.to_ascii_lowercase() as u16;
let w = key as usize >> 6;
let bit = 1u64 << (key as usize & 63);
if seen_consec[w] & bit == 0 {
seen_consec[w] |= bit;
let col = self.get_or_alloc_column(key);
if col != NO_COLUMN {
self.column_bitset(col)[word_idx].fetch_or(bit_mask, Ordering::Relaxed);
}
}
}
// Skip-1 bigram: (bytes[i-2], bytes[i])
let skip_prev = bytes[i - 2];
if (32..=126).contains(&skip_prev) && (32..=126).contains(&cur) {
let key =
(skip_prev.to_ascii_lowercase() as u16) << 8 | cur.to_ascii_lowercase() as u16;
let w = key as usize >> 6;
let bit = 1u64 << (key as usize & 63);
if seen_skip[w] & bit == 0 {
seen_skip[w] |= bit;
let col = skip_builder.get_or_alloc_column(key);
if col != NO_COLUMN {
skip_builder.column_bitset(col)[word_idx]
.fetch_or(bit_mask, Ordering::Relaxed);
}
for &b in &bytes[2..len] {
let cur = normalize_byte_scalar(b);
if cur != u16::MAX {
if n1 != u16::MAX {
let key = (n1 << 8) | cur;
self.record_bigram(&mut seen_consec, key, word_idx, bit_mask);
}
if n0 != u16::MAX {
let key = (n0 << 8) | cur;
skip_builder.record_bigram(&mut seen_skip, key, word_idx, bit_mask);
}
}
n0 = n1;
n1 = cur;
}
self.populated.fetch_add(1, Ordering::Relaxed);
skip_builder.populated.fetch_add(1, Ordering::Relaxed);
}
/// Mark `key` as present for the file whose column-word is `word_idx`
/// and bit position is `bit_mask`, de-duplicating via the caller-owned
/// `seen` bitmap so we only touch the shared column slab at most once
/// per unique bigram per file.
///
/// SAFETY: under the partitioning invariant on `add_file_content`
/// the `word_idx` slot this touches is owned exclusively by the
/// current thread, so a plain `|=` through the raw pointer is
/// race-free (no atomic RMW needed).
#[inline(always)]
fn record_bigram(&self, seen: &mut [u64; 1024], key: u16, word_idx: usize, bit_mask: u64) {
let k = key as usize;
let w = k >> 6;
let bit = 1u64 << (k & 63);
if seen[w] & bit == 0 {
seen[w] |= bit;
let col = self.get_or_alloc_column(key);
if col != NO_COLUMN {
unsafe {
let p = self.column_word_ptr(col, word_idx);
*p |= bit_mask;
}
}
}
}
pub fn is_ready(&self) -> bool {
self.populated.load(Ordering::Relaxed) > 0
}
@@ -157,6 +210,7 @@ impl BigramIndexBuilder {
/// the default ~3.1% heuristic when `None`) and <90% of indexed files.
/// Sparse columns carry too little data to justify their memory;
/// ubiquitous columns (≥90%) are nearly all-ones and barely filter.
#[inline(always)]
pub fn compress(self, min_density_pct: Option<u32>) -> BigramFilter {
let cols = self.columns_used() as usize;
let words = self.words;
@@ -165,58 +219,57 @@ impl BigramIndexBuilder {
let dense_bytes = words * 8; // cost of one dense column
let old_lookup = self.lookup;
let col_data = self.col_data;
// If no file ever populated content, col_data was never
// materialised. Treat as empty — every column falls through.
let col_data: Option<Box<[u64]>> = self.col_data.into_inner().map(UnsafeCell::into_inner);
let mut lookup: Vec<u16> = vec![NO_COLUMN; 65536];
let mut dense_data: Vec<u64> = Vec::with_capacity(cols * words);
let mut dense_count: usize = 0;
for key in 0..65536usize {
let old_col = old_lookup[key].load(Ordering::Relaxed);
if old_col == NO_COLUMN || old_col as usize >= cols {
continue;
}
if let Some(col_data) = col_data.as_deref() {
for key in 0..65536usize {
let old_col = old_lookup[key].load(Ordering::Relaxed);
if old_col == NO_COLUMN || old_col as usize >= cols {
continue;
}
let col_start = old_col as usize * words;
let bitset = &col_data[col_start..col_start + words];
let col_start = old_col as usize * words;
let bitset = &col_data[col_start..col_start + words];
// count set bits to decide if this column is worth keeping.
let mut popcount = 0u32;
for column in bitset.iter().take(words) {
popcount += column.load(Ordering::Relaxed).count_ones();
}
// count set bits to decide if this column is worth keeping.
let mut popcount = 0u32;
for &word in bitset.iter().take(words) {
popcount += word.count_ones();
}
// drop bigrams appearing in too few files
let not_to_rare = if let Some(min_pct) = min_density_pct {
// Percentage-based: require ≥ min_pct% of populated files.
populated > 0 && (popcount as usize) * 100 >= populated * min_pct as usize
} else {
// Default: popcount ≥ words × 2 (~3.1% of files).
(popcount as usize * 4) >= dense_bytes
};
// drop bigrams appearing in too few files
let not_to_rare = if let Some(min_pct) = min_density_pct {
// Percentage-based: require ≥ min_pct% of populated files.
populated > 0 && (popcount as usize) * 100 >= populated * min_pct as usize
} else {
// Default: popcount ≥ words × 2 (~3.1% of files).
(popcount as usize * 4) >= dense_bytes
};
if !not_to_rare {
continue;
}
if !not_to_rare {
continue;
}
// Drop ubiquitous bigrams — columns ≥90% ones carry almost no
// filtering power and just waste memory + AND cycles.
if populated > 0 && (popcount as usize) * 10 >= populated * 9 {
continue;
}
// Drop ubiquitous bigrams — columns ≥90% ones carry almost no
// filtering power and just waste memory + AND cycles.
if populated > 0 && (popcount as usize) * 10 >= populated * 9 {
continue;
}
let dense_idx = dense_count as u16;
lookup[key] = dense_idx;
dense_count += 1;
let dense_idx = dense_count as u16;
lookup[key] = dense_idx;
dense_count += 1;
for column in bitset.iter().take(words) {
dense_data.push(column.load(Ordering::Relaxed));
dense_data.extend_from_slice(bitset);
}
}
// col_data + old_lookup dropped here — single deallocation each,
// no fragmentation.
BigramFilter {
lookup,
dense_data,
@@ -230,7 +283,6 @@ impl BigramIndexBuilder {
}
unsafe impl Send for BigramIndexBuilder {}
unsafe impl Sync for BigramIndexBuilder {}
/// Inverted bigram index with optional "skip-1" extension
/// Copmressed into bitset for minimal usage, the layout of this struct actually matters
@@ -428,6 +480,24 @@ impl BigramFilter {
}
}
/// Map a single input byte to its normalised form used by the bigram
/// builder: `u16::MAX` when not printable ASCII (outside `32..=126`),
/// otherwise the lowercased byte value in `0..=126`. The `u16::MAX`
/// sentinel can never collide with a printable-ASCII byte so the consumer
/// can test `!= u16::MAX` without false positives.
///
/// Branchless and `#[inline(always)]`: LLVM lifts the ASCII-range check
/// and the conditional-lowercase OR into a handful of instructions per
/// call, so calling this inside a hot loop matches a hand-unrolled
/// equivalent.
#[inline(always)]
fn normalize_byte_scalar(b: u8) -> u16 {
let printable = b.wrapping_sub(32) <= 94;
// Branchless lowercase: OR 0x20 iff byte is in 'A'..='Z'.
let lower = b | ((b.wrapping_sub(b'A') < 26) as u8 * 0x20);
if printable { lower as u16 } else { u16::MAX }
}
pub fn extract_bigrams(content: &[u8]) -> Vec<u16> {
if content.len() < 2 {
return Vec::new();
@@ -525,3 +595,547 @@ impl BigramOverlay {
self.modified.keys().copied().collect()
}
}
pub const BIGRAM_CONTENT_CAP: usize = 64 * 1024;
const BIGRAM_CHUNK_FILES: usize = 4 * 64;
/// Sparse-column cutoff for the skip-1 sub-index. Rare skip columns add
/// little filtering power but ~25-30% of index memory, so we drop
/// anything appearing in < 12 % of populated files.
const SKIP_INDEX_MIN_DENSITY_PCT: u32 = 12;
thread_local! {
/// Per-rayon-worker reusable read buffer. 64 KB is too large to
/// keep on the default pthread stack (macOS ships 512 KB), so the
/// buffer lives on the heap behind a `Box<[u8; N]>`. TLS keeps the
/// allocation alive for the thread's lifetime so we pay the cost
/// once, not per file.
static READ_BUF: std::cell::RefCell<Box<[u8; BIGRAM_CONTENT_CAP]>> =
std::cell::RefCell::new(Box::new([0u8; BIGRAM_CONTENT_CAP]));
}
/// Outcome of processing one file's content.
enum FileOutcome {
/// Content contained a NUL byte — mark the file as binary so future
/// greps skip it without re-reading.
Binary,
/// Read succeeded and the content was fed to the bigram builder.
Indexed,
/// File was empty or failed to open; nothing to do.
Skipped,
}
#[tracing::instrument(skip_all, name = "Building Bigram Index", level = tracing::Level::DEBUG)]
pub(crate) fn build_bigram_index(
files: &[crate::types::FileItem],
budget: &crate::types::ContentCacheBudget,
base_path: &std::path::Path,
arena: crate::simd_path::ArenaPtr,
) -> (BigramFilter, Vec<usize>) {
let start = std::time::Instant::now();
tracing::info!("Building bigram index for {} files...", files.len());
let builder = BigramIndexBuilder::new(files.len());
let skip_builder = BigramIndexBuilder::new(files.len());
// this does remove a memcpy for every single file + actually reducing open time on macos
#[cfg(unix)]
let base_fd: libc::c_int = open_base_dir_fd(base_path);
#[cfg(not(unix))]
let base_fd: i32 = -1;
// `content_binary` is only touched from the Binary branch below, so
// the mutex is cold in practice. A lock-free collector wasn't worth
// the complexity.
let content_binary: std::sync::Mutex<Vec<usize>> = std::sync::Mutex::new(Vec::new());
crate::file_picker::BACKGROUND_THREAD_POOL.install(|| {
files
.par_chunks(BIGRAM_CHUNK_FILES)
.enumerate()
.for_each(|(chunk_idx, chunk)| {
let base_idx = chunk_idx * BIGRAM_CHUNK_FILES;
for (offset, file) in chunk.iter().enumerate() {
let file_idx = base_idx + offset;
let outcome = process_file(
file,
file_idx,
&builder,
&skip_builder,
base_fd,
base_path,
arena,
budget,
);
if matches!(outcome, FileOutcome::Binary) {
content_binary.lock().unwrap().push(file_idx);
}
}
});
});
#[cfg(unix)]
if base_fd >= 0 {
// SAFETY: we opened `base_fd` at the top of this function and
// no worker still references it once the rayon pool joined.
unsafe { libc::close(base_fd) };
}
let content_binary_vec = content_binary.into_inner().unwrap();
let cols = builder.columns_used();
let mut index = builder.compress(None);
let skip_index = skip_builder.compress(Some(SKIP_INDEX_MIN_DENSITY_PCT));
index.set_skip_index(skip_index);
// Builder buffers were freed by `compress()` above (one deallocation
// each); nudge mimalloc to return them (and any transient allocs)
// to the OS.
crate::file_picker::hint_allocator_collect();
tracing::info!(
"Bigram index built in {:.2}s — {} dense columns for {} files",
start.elapsed().as_secs_f64(),
cols,
files.len(),
);
if !content_binary_vec.is_empty() {
tracing::info!(
"Bigram build detected {} content-binary files (not caught by extension)",
content_binary_vec.len(),
);
}
(index, content_binary_vec)
}
/// Process one file: read up to `BIGRAM_CONTENT_CAP` bytes, feed them
/// to the bigram builder (or record as binary / skipped).
///
/// `base_fd` is the parent-directory fd for the Unix `openat` fast
/// path, or `-1` to force the portable `std::fs::File::open` fallback.
#[inline]
#[allow(clippy::too_many_arguments)]
fn process_file(
file: &crate::types::FileItem,
file_idx: usize,
builder: &BigramIndexBuilder,
skip_builder: &BigramIndexBuilder,
base_fd: i32,
base_path: &std::path::Path,
arena: crate::simd_path::ArenaPtr,
budget: &crate::types::ContentCacheBudget,
) -> FileOutcome {
if file.is_binary() || file.size == 0 || file.size > budget.max_file_size {
return FileOutcome::Skipped;
}
// Zero-copy fast path: the warmup phase may have cached this file's
// content already. Avoid re-reading from disk.
if let Some(cached) = file.get_content(arena, base_path, budget) {
if crate::file_picker::detect_binary_content(cached) {
return FileOutcome::Binary;
}
let capped = &cached[..cached.len().min(BIGRAM_CONTENT_CAP)];
builder.add_file_content(skip_builder, file_idx, capped);
return FileOutcome::Indexed;
}
let want = (file.size as usize).min(BIGRAM_CONTENT_CAP);
let mut path_buf = [0u8; crate::simd_path::PATH_BUF_SIZE];
READ_BUF.with(|read_cell| {
let mut buf = read_cell.borrow_mut();
let filled = read_file_content(
file,
base_fd,
base_path,
arena,
&mut path_buf,
&mut buf[..want],
);
if filled == 0 {
return FileOutcome::Skipped;
}
let data = &buf[..filled];
if crate::file_picker::detect_binary_content(data) {
return FileOutcome::Binary;
}
builder.add_file_content(skip_builder, file_idx, data);
FileOutcome::Indexed
})
}
/// Read up to `buf.len()` bytes of `file`'s content into `buf`. Returns
/// the number of bytes actually read (0 on any error, so callers treat
/// failures as "skip").
#[inline]
fn read_file_content(
file: &crate::types::FileItem,
base_fd: i32,
base_path: &std::path::Path,
arena: crate::simd_path::ArenaPtr,
path_buf: &mut [u8; crate::simd_path::PATH_BUF_SIZE],
buf: &mut [u8],
) -> usize {
#[cfg(unix)]
{
read_file_content_unix(file, base_fd, base_path, arena, path_buf, buf)
}
#[cfg(not(unix))]
{
let _ = base_fd;
read_file_content_std(file, base_path, arena, path_buf, buf)
}
}
#[cfg(unix)]
fn read_file_content_unix(
file: &crate::types::FileItem,
base_fd: libc::c_int,
base_path: &std::path::Path,
arena: crate::simd_path::ArenaPtr,
path_buf: &mut [u8; crate::simd_path::PATH_BUF_SIZE],
buf: &mut [u8],
) -> usize {
let fd = if base_fd >= 0 {
let rel_cstr = file.write_relative_cstr(arena, path_buf);
// SAFETY: `rel_cstr` is NUL-terminated, `base_fd` is a valid
// directory descriptor owned by the caller.
unsafe { libc::openat(base_fd, rel_cstr.as_ptr(), libc::O_RDONLY) }
} else {
use std::os::unix::io::IntoRawFd;
let abs = file.write_absolute_path(arena, base_path, path_buf);
match std::fs::File::open(abs) {
Ok(f) => f.into_raw_fd(),
Err(_) => return 0,
}
};
if fd < 0 {
return 0;
}
let mut filled = 0usize;
while filled < buf.len() {
// SAFETY: `fd` is an owned descriptor, `buf[filled..]` is a
// valid writable slice for `buf.len() - filled` bytes.
let n = unsafe {
libc::read(
fd,
buf[filled..].as_mut_ptr() as *mut libc::c_void,
(buf.len() - filled) as libc::size_t,
)
};
if n <= 0 {
break;
}
filled += n as usize;
}
// SAFETY: matching close for the owned descriptor.
unsafe { libc::close(fd) };
filled
}
/// Open the base directory for the `openat` fast path. Returns `-1` on
/// failure — callers interpret a negative fd as "fall back to absolute
/// paths".
#[cfg(unix)]
fn open_base_dir_fd(base_path: &std::path::Path) -> libc::c_int {
use std::os::unix::ffi::OsStrExt;
let mut cstr = [0u8; crate::simd_path::PATH_BUF_SIZE];
let bytes = base_path.as_os_str().as_bytes();
if bytes.len() >= cstr.len() {
return -1;
}
cstr[..bytes.len()].copy_from_slice(bytes);
// SAFETY: `cstr` is NUL-terminated by construction (zero-initialised,
// and we only filled up to `bytes.len() < cstr.len()`).
unsafe {
libc::open(
cstr.as_ptr() as *const std::os::raw::c_char,
libc::O_RDONLY | libc::O_DIRECTORY,
)
}
}
/// Portable fallback (Windows + non-`openat` Unix): `std::fs::File` +
/// `Read::read` into `buf`. Used on Windows unconditionally, and on
/// Unix when the base directory fd could not be opened.
#[cfg(not(unix))]
fn read_file_content_std(
file: &crate::types::FileItem,
base_path: &std::path::Path,
arena: crate::simd_path::ArenaPtr,
path_buf: &mut [u8; crate::simd_path::PATH_BUF_SIZE],
buf: &mut [u8],
) -> usize {
use std::io::Read;
let abs = file.write_absolute_path(arena, base_path, path_buf);
let Ok(mut f) = std::fs::File::open(abs) else {
return 0;
};
let mut filled = 0usize;
while filled < buf.len() {
match f.read(&mut buf[filled..]) {
Ok(0) => break,
Ok(n) => filled += n,
Err(_) => return 0,
}
}
filled
}
#[cfg(test)]
mod tests {
use super::*;
/// Build a key the same way `add_file_content` does: two printable-ASCII
/// bytes, lowercased, packed as `(hi << 8) | lo`.
fn key(a: u8, b: u8) -> u16 {
((a.to_ascii_lowercase() as u16) << 8) | b.to_ascii_lowercase() as u16
}
/// Return the sorted list of (consec, skip) bigram keys that should appear
/// for `content`. Used as the reference implementation.
fn expected_bigrams(content: &[u8]) -> (Vec<u16>, Vec<u16>) {
let mut consec: std::collections::BTreeSet<u16> = Default::default();
let mut skip: std::collections::BTreeSet<u16> = Default::default();
let printable = |b: u8| (32..=126).contains(&b);
for i in 1..content.len() {
let a = content[i - 1];
let b = content[i];
if printable(a) && printable(b) {
consec.insert(key(a, b));
}
if i >= 2 {
let a = content[i - 2];
let b = content[i];
if printable(a) && printable(b) {
skip.insert(key(a, b));
}
}
}
(consec.into_iter().collect(), skip.into_iter().collect())
}
/// Query: does the builder record file 0 as having this bigram set?
fn builder_has_key_for_file_0(b: &BigramIndexBuilder, k: u16) -> bool {
let col = b.lookup[k as usize].load(Ordering::Relaxed);
if col == NO_COLUMN {
return false;
}
b.column_bitset(col)[0] & 1 != 0
}
fn run_and_compare(content: &[u8]) {
let consec = BigramIndexBuilder::new(1);
let skip = BigramIndexBuilder::new(1);
consec.add_file_content(&skip, 0, content);
let (expected_consec, expected_skip) = expected_bigrams(content);
// Every expected bigram must be recorded.
for k in &expected_consec {
assert!(
builder_has_key_for_file_0(&consec, *k),
"consec bigram 0x{k:04x} missing for content {content:?}",
);
}
for k in &expected_skip {
assert!(
builder_has_key_for_file_0(&skip, *k),
"skip bigram 0x{k:04x} missing for content {content:?}",
);
}
// No unexpected bigrams — iterate lookup for set columns.
for k in 0u32..=0xFFFF {
let recorded_consec = builder_has_key_for_file_0(&consec, k as u16);
let recorded_skip = builder_has_key_for_file_0(&skip, k as u16);
if recorded_consec {
assert!(
expected_consec.contains(&(k as u16)),
"unexpected consec bigram 0x{k:04x} in content {content:?}",
);
}
if recorded_skip {
assert!(
expected_skip.contains(&(k as u16)),
"unexpected skip bigram 0x{k:04x} in content {content:?}",
);
}
}
}
#[test]
fn add_file_empty_is_noop() {
let consec = BigramIndexBuilder::new(1);
let skip = BigramIndexBuilder::new(1);
consec.add_file_content(&skip, 0, b"");
assert_eq!(consec.columns_used(), 0);
assert_eq!(skip.columns_used(), 0);
// populated counter not incremented for empty input
assert_eq!(consec.populated.load(Ordering::Relaxed), 0);
}
#[test]
fn add_file_single_byte_is_noop() {
let consec = BigramIndexBuilder::new(1);
let skip = BigramIndexBuilder::new(1);
consec.add_file_content(&skip, 0, b"a");
assert_eq!(consec.columns_used(), 0);
assert_eq!(skip.columns_used(), 0);
}
#[test]
fn add_file_two_bytes_consec_only() {
// With exactly 2 bytes there's no skip bigram (needs i >= 2 in the loop).
run_and_compare(b"ab");
}
#[test]
fn add_file_three_bytes_has_skip() {
// "abc" -> consec {"ab", "bc"}, skip {"ac"}
run_and_compare(b"abc");
}
#[test]
fn add_file_ascii_words() {
run_and_compare(b"hello world");
run_and_compare(b"the quick brown fox jumps over the lazy dog");
run_and_compare(b"fn main() { println!(\"hi\"); }");
}
#[test]
fn add_file_case_is_lowered() {
// Uppercase should be lowercased before keying, so "AB" == "ab".
let upper = BigramIndexBuilder::new(1);
let upper_skip = BigramIndexBuilder::new(1);
upper.add_file_content(&upper_skip, 0, b"ABC");
let lower = BigramIndexBuilder::new(1);
let lower_skip = BigramIndexBuilder::new(1);
lower.add_file_content(&lower_skip, 0, b"abc");
// Both should have identical bigram keys.
for k in 0u32..=0xFFFF {
let u = builder_has_key_for_file_0(&upper, k as u16);
let l = builder_has_key_for_file_0(&lower, k as u16);
assert_eq!(u, l, "consec 0x{k:04x}: upper={u} lower={l}");
let u = builder_has_key_for_file_0(&upper_skip, k as u16);
let l = builder_has_key_for_file_0(&lower_skip, k as u16);
assert_eq!(u, l, "skip 0x{k:04x}: upper={u} lower={l}");
}
}
#[test]
fn add_file_rejects_non_printable() {
// Bigrams where either byte is outside 32..=126 are rejected. But
// the skip-1 bigram can still connect two printable bytes across a
// non-printable one: for "\0a\0b", consec sees no valid pair but
// skip sees (a,b) at i=3. Use the reference implementation.
run_and_compare(b"\0a\0b");
// All-zero input: truly nothing recorded.
let consec = BigramIndexBuilder::new(1);
let skip = BigramIndexBuilder::new(1);
consec.add_file_content(&skip, 0, b"\0\0\0\0");
assert_eq!(consec.columns_used(), 0);
assert_eq!(skip.columns_used(), 0);
}
#[test]
fn add_file_mixed_printable_and_control() {
// "a\tb\nc d" — \t (9) and \n (10) are below 32. Consec:
// (a, \t) x, (\t, b) x, (b, \n) x, (\n, c) x, (c, ' ') ok, (' ', d) ok
// Skip (i-2, i):
// (a, b) ok, (\t, \n) x, (b, c) ok, (\n, ' ') x, (c, d) ok
run_and_compare(b"a\tb\nc d");
}
#[test]
fn add_file_repeats_are_deduped() {
// "ababab" has many repeats of "ab", "ba" — each unique bigram should
// be recorded exactly once (the stack-local `seen_*` dedup works).
run_and_compare(b"ababababab");
}
#[test]
fn add_file_tombstone_separation() {
// Two separate files share no bits; file 1's content doesn't bleed
// into file 0's row and vice-versa.
let consec = BigramIndexBuilder::new(2);
let skip = BigramIndexBuilder::new(2);
consec.add_file_content(&skip, 0, b"xy");
consec.add_file_content(&skip, 1, b"zw");
let key_xy = key(b'x', b'y');
let key_zw = key(b'z', b'w');
// file 0 has "xy" but not "zw"
let col_xy = consec.lookup[key_xy as usize].load(Ordering::Relaxed);
let col_zw = consec.lookup[key_zw as usize].load(Ordering::Relaxed);
let bitset_xy = consec.column_bitset(col_xy)[0];
let bitset_zw = consec.column_bitset(col_zw)[0];
assert_eq!(bitset_xy & 0b01, 0b01, "file 0 should have xy");
assert_eq!(bitset_zw & 0b01, 0, "file 0 should NOT have zw");
assert_eq!(bitset_xy & 0b10, 0, "file 1 should NOT have xy");
assert_eq!(bitset_zw & 0b10, 0b10, "file 1 should have zw");
}
#[test]
fn add_file_long_content() {
// Stress test: ~8 KB of printable ASCII. Should complete without
// overflowing any stack-local bitset and produce the full set.
let mut buf = Vec::with_capacity(8192);
for i in 0..8192 {
buf.push(32u8 + ((i * 7) % 95) as u8); // cycle through printable range
}
run_and_compare(&buf);
}
#[test]
fn add_file_simd_and_scalar_agree() {
// Cross-check: both code paths (scalar <128 bytes, SIMD ≥128) must
// produce identical bigram sets for content that straddles the
// threshold. Mix printable ASCII with some non-printable bytes and
// repeats so the non-printable branch in the SIMD path exercises.
let mut mixed = Vec::with_capacity(256);
for i in 0..256usize {
mixed.push(match i % 9 {
0 => 0, // NUL
1 => 0x7F, // DEL (just above 126)
2 => b'\n', // below 32
_ => 32 + ((i * 13) % 95) as u8,
});
}
run_and_compare(&mixed[..127]); // scalar path
run_and_compare(&mixed); // SIMD path (256 bytes)
run_and_compare(&mixed[..192]); // SIMD path with scalar tail
}
#[test]
fn add_file_respects_file_count_boundary() {
// file_count=100, file_idx=63 (last bit in word 0) and file_idx=64
// (first bit in word 1). Make sure the word_idx math is right.
let consec = BigramIndexBuilder::new(100);
let skip = BigramIndexBuilder::new(100);
consec.add_file_content(&skip, 63, b"ab");
consec.add_file_content(&skip, 64, b"cd");
let kab = key(b'a', b'b');
let kcd = key(b'c', b'd');
let col_ab = consec.lookup[kab as usize].load(Ordering::Relaxed);
let col_cd = consec.lookup[kcd as usize].load(Ordering::Relaxed);
let ab_bitset = consec.column_bitset(col_ab);
let cd_bitset = consec.column_bitset(col_cd);
// ab in word 0, bit 63
assert_eq!(ab_bitset[0], 1u64 << 63);
assert_eq!(ab_bitset[1], 0);
// cd in word 1, bit 0
assert_eq!(cd_bitset[0], 0);
assert_eq!(cd_bitset[1], 1);
}
}
crates/fff-core/src/error.rs
+1 -1
View File
@@ -8,7 +8,7 @@ pub enum Error {
#[error("Invalid path {0}")]
InvalidPath(std::path::PathBuf),
#[error(
"Can not start fff at the file system root {0} — pass a project or at least home directory instead"
"Can not run certain FFF features in a file system root or home directories. Consider smaller per-project directories."
)]
FilesystemRoot(std::path::PathBuf),
#[error("File picker not initialized")]
crates/fff-core/src/file_picker.rs
+789 -1014
View File
File diff suppressed because it is too large Load Diff
crates/fff-core/src/git.rs
+110 -26
View File
@@ -1,4 +1,5 @@
use crate::error::Result;
use ahash::AHashMap;
use git2::{Repository, Status, StatusOptions};
use std::{
fmt::Debug,
@@ -6,15 +7,21 @@ use std::{
};
use tracing::debug;
/// Represents a cache of a single git status query, if there is no
/// status aka file is clear but it was specifically requested to updated
/// the status is `None` otherwise contains only actual file statuses.
#[derive(Debug, Clone)]
pub struct GitStatusCache(Vec<(PathBuf, Status)>);
pub(crate) fn default_status_options() -> StatusOptions {
let mut opts = StatusOptions::new();
opts.include_untracked(true)
.recurse_untracked_dirs(true)
.include_unmodified(true)
.exclude_submodules(true);
opts
}
#[derive(Debug, Clone, Default)]
pub(crate) struct GitStatusCache(AHashMap<PathBuf, Status>);
impl IntoIterator for GitStatusCache {
type Item = (PathBuf, Status);
type IntoIter = std::vec::IntoIter<Self::Item>;
type IntoIter = <AHashMap<PathBuf, Status> as IntoIterator>::IntoIter;
fn into_iter(self) -> Self::IntoIter {
self.0.into_iter()
@@ -26,25 +33,27 @@ impl GitStatusCache {
self.0.len()
}
#[inline]
pub fn lookup_status(&self, full_path: &Path) -> Option<Status> {
self.0
.binary_search_by(|(path, _)| path.as_path().cmp(full_path))
.ok()
.and_then(|idx| self.0.get(idx).map(|(_, status)| *status))
self.0.get(full_path).copied()
}
#[tracing::instrument(skip(repo, status_options))]
fn read_status_impl(repo: &Repository, status_options: &mut StatusOptions) -> Result<Self> {
let statuses = repo.statuses(Some(status_options))?;
let Some(repo_path) = repo.workdir() else {
return Ok(Self(vec![])); // repo is bare
return Ok(Self(AHashMap::new())); // repo is bare
};
let mut entries = Vec::with_capacity(statuses.len());
let repo_path = crate::path_utils::normalize(repo_path.to_path_buf());
let mut entries = AHashMap::with_capacity(statuses.len());
for entry in &statuses {
if let Some(entry_path) = entry.path() {
let full_path = repo_path.join(entry_path);
entries.push((full_path, entry.status()));
// libgit2 returns entry paths with forward slashes on every platform
// fff stores native paths - meaning we have forward slash issue on windows
let full_path = crate::path_utils::normalize(repo_path.join(entry_path));
entries.insert(full_path, entry.status());
}
}
@@ -76,32 +85,29 @@ impl GitStatusCache {
paths: &[TPath],
) -> Result<Self> {
if paths.is_empty() {
return Ok(Self(vec![]));
return Ok(Self(AHashMap::new()));
}
let Some(workdir) = repo.workdir() else {
return Ok(Self(vec![]));
return Ok(Self(AHashMap::new()));
};
let workdir = crate::path_utils::normalize(workdir.to_path_buf());
// git pathspec is pretty slow and requires to walk the whole directory
// so for a single file which is the most general use case we query directly the file
if paths.len() == 1 {
let full_path = paths[0].as_ref();
let relative_path = full_path.strip_prefix(workdir)?;
let relative_path = full_path.strip_prefix(&workdir)?;
let status = repo.status_file(relative_path)?;
return Ok(Self(vec![(full_path.to_path_buf(), status)]));
let mut map = AHashMap::with_capacity(1);
map.insert(full_path.to_path_buf(), status);
return Ok(Self(map));
}
let mut status_options = StatusOptions::new();
status_options
.include_untracked(true)
.recurse_untracked_dirs(true)
// when reading partial status it's important to include all files requested
.include_unmodified(true);
let mut status_options = default_status_options();
for path in paths {
status_options.pathspec(path.as_ref().strip_prefix(workdir)?);
status_options.pathspec(path.as_ref().strip_prefix(&workdir)?);
}
let git_status_cache = Self::read_status_impl(repo, &mut status_options)?;
@@ -157,3 +163,81 @@ pub fn format_git_status_opt(status: Option<Status>) -> Option<&'static str> {
pub fn format_git_status(status: Option<Status>) -> &'static str {
format_git_status_opt(status).unwrap_or("unknown")
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
use std::process::Command;
use tempfile::TempDir;
fn git(dir: &Path, args: &[&str]) {
let out = Command::new("git")
.args(args)
.current_dir(dir)
.env("GIT_AUTHOR_NAME", "t")
.env("GIT_AUTHOR_EMAIL", "t@t")
.env("GIT_COMMITTER_NAME", "t")
.env("GIT_COMMITTER_EMAIL", "t@t")
.output()
.unwrap();
assert!(out.status.success(), "git {args:?} failed");
}
/// Regression: on case-insensitive filesystems libgit2 returns
/// statuses in a case-insensitive order. Our previous sorted-`Vec` +
/// `binary_search_by(Path::cmp)` lookup silently missed entries
/// because `Path::cmp` is byte-wise.
///
/// This test uses deliberately mixed-case filenames so the two
/// orderings disagree, then checks every lookup succeeds.
#[test]
fn lookup_is_case_exact_regardless_of_libgit2_sort_order() {
let tmp = TempDir::new().unwrap();
let base = tmp.path().canonicalize().unwrap();
// Mixed-case names that sort differently under byte-wise vs
// case-insensitive comparators.
let names = [
"README.md",
"a_lower.rs",
"Z_upper.rs",
"mixed_Case.txt",
"nested/Inner_File.rs",
];
for n in &names {
let p = base.join(n);
fs::create_dir_all(p.parent().unwrap()).unwrap();
fs::write(&p, format!("// {n}\n")).unwrap();
}
git(&base, &["init", "-b", "main"]);
git(&base, &["add", "-A"]);
git(&base, &["commit", "-m", "seed", "--no-gpg-sign"]);
// Modify every file so they all end up in the status output as
// WT_MODIFIED — guarantees a non-trivial map we have to look up.
for n in &names {
let p = base.join(n);
fs::write(&p, format!("// {n}\n// edit\n")).unwrap();
}
let repo = Repository::open(&base).unwrap();
let paths: Vec<PathBuf> = names.iter().map(|n| base.join(n)).collect();
let cache = GitStatusCache::git_status_for_paths(&repo, &paths).unwrap();
for (n, abs) in names.iter().zip(paths.iter()) {
let status = cache.lookup_status(abs);
assert!(
status.is_some(),
"lookup for {n} returned None; cache holds {} entries",
cache.statuses_len(),
);
assert!(
status.unwrap().contains(Status::WT_MODIFIED),
"expected WT_MODIFIED for {n}, got {:?}",
status
);
}
}
}
crates/fff-core/src/grep.rs
+25 -1
View File
@@ -1023,14 +1023,24 @@ pub(crate) fn multi_grep_search<'a>(
filtered_file_count = retry_count;
}
// Apply bigram prefilter to the file list
// Apply bigram prefilter to the file list. Bigram columns only cover
// the indexable region — `bigram_overlay.base_file_count()` is the
// authoritative boundary. Files past it (unindexable base, overflow)
// are always retained and searched directly.
if let Some(ref candidates) = bigram_candidates {
let base_ptr = files.as_ptr();
let bigram_boundary = bigram_overlay
.map(|o| o.base_file_count())
.unwrap_or(files.len());
files_to_search.retain(|f| {
if f.is_overflow() {
return true;
}
let file_idx = unsafe { (*f as *const FileItem).offset_from(base_ptr) as usize };
if file_idx >= bigram_boundary {
return true;
}
BigramFilter::is_candidate(candidates, file_idx)
});
}
@@ -1941,7 +1951,17 @@ pub(crate) fn grep_search<'a>(
}
}
// Bigram columns only cover the indexable region at
// `files[..overlay.base_file_count()]` — which equals
// `indexable_count` today. Files past that boundary
// (unindexable base files, overflow) are not tracked by
// the bigram filter, so we always retain them and let
// the full text search decide.
let base_ptr = files.as_ptr();
let bigram_boundary = bigram_overlay
.map(|o| o.base_file_count())
.unwrap_or(files.len());
files_to_search.retain(|f| {
if f.is_overflow() {
return true;
@@ -1950,6 +1970,10 @@ pub(crate) fn grep_search<'a>(
let file_idx =
unsafe { (*f as *const FileItem).offset_from(base_ptr) as usize };
if file_idx >= bigram_boundary {
return true;
}
BigramFilter::is_candidate(&candidates, file_idx)
});
}
crates/fff-core/src/ignore.rs
+10 -2
View File
@@ -14,8 +14,16 @@ pub(crate) const NON_GIT_IGNORED_DIRS: &[&str] = &[
];
#[cfg(target_os = "macos")]
pub(crate) const PLATFORM_IGNORED_DIRS: &[&str] =
&["Library/Application Support", "Library/Caches"];
pub(crate) const PLATFORM_IGNORED_DIRS: &[&str] = &[
"Library/Application Support",
"Library/Caches",
// App-group sandbox storage — used by iMessage, Photos, Notes, Calendar,
// Electron apps, etc. for SQLite-WAL, LevelDB, protobuf files. These are
// almost entirely extension-less binary files (~80k on a typical $HOME)
// that never need to appear in a fuzzy or grep search.
"Library/Group Containers",
"Library/Containers",
];
#[cfg(target_os = "windows")]
pub(crate) const PLATFORM_IGNORED_DIRS: &[&str] = &[
crates/fff-core/src/lib.rs
+8 -4
View File
@@ -20,7 +20,7 @@
//!
//! ## Shared State
//!
//! [`SharedPicker`], [`SharedFrecency`], and [`SharedQueryTracker`] are
//! [`SharedFilePicker`], [`SharedFrecency`], and [`SharedQueryTracker`] are
//! newtype wrappers around `Arc<RwLock<Option<T>>>` for thread-safe shared
//! access. They provide `read()` / `write()` methods with built-in error
//! conversion and convenience helpers like `wait_for_scan()`.
@@ -33,10 +33,10 @@
//! use fff_search::query_tracker::QueryTracker;
//! use fff_search::{
//! FFFMode, FilePickerOptions, FuzzySearchOptions, PaginationArgs, QueryParser,
//! SharedFrecency, SharedPicker, SharedQueryTracker,
//! SharedFrecency, SharedFilePicker, SharedQueryTracker,
//! };
//!
//! let shared_picker = SharedPicker::default();
//! let shared_picker = SharedFilePicker::default();
//! let shared_frecency = SharedFrecency::default();
//! let shared_query_tracker = SharedQueryTracker::default();
//!
@@ -92,7 +92,11 @@
//! ```
mod background_watcher;
mod bigram_filter;
mod scan;
// public only for benchmarks — the inverted index is still re-exported via
// `pub use bigram_filter::*` below for external consumers.
#[doc(hidden)]
pub mod bigram_filter;
pub mod bigram_query;
mod constraints;
mod db_healthcheck;
crates/fff-core/src/path_utils.rs
+14
View File
@@ -10,6 +10,20 @@ pub fn canonicalize(path: impl AsRef<Path>) -> std::io::Result<PathBuf> {
std::fs::canonicalize(path)
}
/// Git requires a normalized forward-slashed paths on windows
#[cfg(windows)]
pub fn normalize(path: PathBuf) -> PathBuf {
let as_str = path.to_string_lossy();
let with_backslashes: String = as_str.replace('/', "\\");
let buf = PathBuf::from(with_backslashes);
dunce::canonicalize(&buf).unwrap_or(buf)
}
#[cfg(not(windows))]
pub fn normalize(path: PathBuf) -> PathBuf {
path
}
#[cfg(windows)]
pub fn expand_tilde(path: &str) -> PathBuf {
return PathBuf::from(path);
crates/fff-core/src/scan.rs
+441
View File
@@ -0,0 +1,441 @@
//! Unified scan-phase orchestrator.
//!
//! Every (re)index code path — initial scan, FFI-triggered rescan,
//! watcher overflow rescan — goes through [`ScanJob::run`]. The
//! orchestrator owns the *sequence* of a scan:
//!
//! 1. walk filesystem off-lock
//! 2. swap `sync_data` under a brief write
//! 3. apply git status + frecency off-lock
//! 4. (optional, initial scan only) spawn the filesystem watcher
//! 5. (optional) post-scan: auto-size cache budget, warmup, bigram
//!
//! The picker write lock is held only in step 2 and step 5's index
//! install — both O(µs-ms), never seconds. Every other FFI caller on
//! the nvim main thread keeps running.
//!
//! ## Entry points
//!
//! - [`ScanJob::spawn`] — fire-and-forget from `SharedPicker` state.
//! Used by the watcher overflow path and by FFI (`scan_files`).
//! - [`ScanJob::spawn_initial`] — same, but takes explicit config for
//! the very first scan, before the `FilePicker` struct lives inside
//! the shared handle.
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use tracing::{error, info};
use crate::FileSync;
use crate::background_watcher::BackgroundWatcher;
use crate::bigram_filter::BigramOverlay;
use crate::bigram_filter::build_bigram_index;
use crate::error::Error;
use crate::file_picker::{self, FFFMode, warmup_mmaps};
use crate::shared::{SharedFilePicker, SharedFrecency};
use crate::types::ContentCacheBudget;
/// Shared atomic flags surfaced by the picker for the scan worker to
/// signal its progress. Grouped so every callsite passes one value,
/// not four.
#[derive(Clone, Default)]
pub(crate) struct ScanSignals {
/// Set to `true` while any scan phase is running
pub(crate) scanning: Arc<AtomicBool>,
/// Set to `true` once the filesystem watcher has been installed
pub(crate) watcher_ready: Arc<AtomicBool>,
/// Indicates that that owning picker was requested to shut down
pub(crate) cancelled: Arc<AtomicBool>,
/// Soft lock indicating that the post scan non blocking work is active
pub(crate) post_scan_busy: Arc<AtomicBool>,
/// Used to resolve conflicts if multiple rescans were triggered in a queue
pub(crate) rescan_pending: Arc<AtomicBool>,
}
/// Which optional phases a scan should run.
#[derive(Clone, Copy, Default)]
pub(crate) struct ScanConfig {
pub(crate) warmup: bool,
pub(crate) content_indexing: bool,
pub(crate) watch: bool,
pub(crate) auto_cache_budget: bool,
pub(crate) install_watcher: bool,
}
/// A fully-configured scan job ready to run on a background thread.
///
/// Build with [`ScanJob::from_picker`] (reads all state from the
/// current `FilePicker`) or [`ScanJob::initial`] (for the bootstrap
/// scan, before the picker is published to `SharedPicker`).
pub(crate) struct ScanJob {
shared_picker: SharedFilePicker,
shared_frecency: SharedFrecency,
base_path: PathBuf,
mode: FFFMode,
signals: ScanSignals,
config: ScanConfig,
/// Walker-maintained counter backing `get_scan_progress` on the UI
/// side. Reset to 0 at scan start, incremented per-file by the
/// walker. Shared `Arc` so the UI polls the same atomic.
scanned_files_counter: Arc<AtomicUsize>,
}
impl ScanJob {
pub fn new(
shared_picker: &SharedFilePicker,
shared_frecency: &SharedFrecency,
install_watcher: bool,
) -> Result<Option<Self>, Error> {
let guard = shared_picker.read()?;
let picker = guard.as_ref().ok_or(Error::FilePickerMissing)?;
if picker.is_scan_active() {
return Ok(None);
}
let signals = picker.scan_signals();
if signals.post_scan_busy.load(Ordering::Acquire) {
return Ok(None);
}
Ok(Some(Self {
shared_picker: shared_picker.clone(),
shared_frecency: shared_frecency.clone(),
base_path: picker.base_path().to_path_buf(),
mode: picker.mode(),
signals,
scanned_files_counter: picker.scanned_files_counter(),
config: ScanConfig {
warmup: picker.has_mmap_cache(),
content_indexing: picker.has_content_indexing(),
watch: picker.has_watcher(),
auto_cache_budget: !picker.has_explicit_cache_budget(),
install_watcher,
},
}))
}
/// Same as [`new`] but without reading from the picker — caller
/// supplies the base path / mode / flags directly. Used by the
/// bootstrap scan before the `FilePicker` is published to
/// `SharedPicker`.
pub fn new_initial(
shared_picker: SharedFilePicker,
shared_frecency: SharedFrecency,
base_path: PathBuf,
mode: FFFMode,
signals: ScanSignals,
scanned_files_counter: Arc<AtomicUsize>,
config: ScanConfig,
) -> Self {
Self {
shared_picker,
shared_frecency,
base_path,
mode,
signals,
scanned_files_counter,
config,
}
}
/// Spawn the job on a dedicated OS thread. Returns immediately.
pub fn spawn(self) -> std::thread::JoinHandle<()> {
self.signals.scanning.store(true, Ordering::Release);
std::thread::Builder::new()
.name("fff-scan".into())
.spawn(move || self.run())
.expect("failed to spawn fff-scan thread")
}
fn run(self) {
let Self {
shared_picker,
shared_frecency,
base_path,
mode,
signals,
scanned_files_counter,
config,
} = self;
let _scanning = ScanningGuard::new(&signals, config.install_watcher);
// Reset the UI-visible counter; the walker bumps it per file
// and `get_scan_progress` reads it without locks.
scanned_files_counter.store(0, Ordering::Relaxed);
// 1. Start git discovery and walk filesystem off-lock.
let git_workdir = FileSync::discover_git_workdir(&base_path);
let status_handle = git_workdir.clone().map(FileSync::spawn_git_status);
let sync = match FileSync::walk_filesystem(
&base_path,
git_workdir,
&scanned_files_counter,
&shared_frecency,
mode,
) {
Ok(sync) => sync,
Err(e) => {
error!(?e, "scan walk failed");
return;
}
};
if signals.cancelled.load(Ordering::Acquire) {
info!("walk completed but picker was replaced, discarding results");
return;
}
let git_workdir = sync.git_workdir.clone();
// 2. Brief write to install the freshly-walked file list.
if let Ok(mut guard) = shared_picker.write()
&& let Some(picker) = guard.as_mut()
{
picker.commit_new_sync(sync);
} else {
error!("failed to install scan results into picker");
return;
}
// Files are now searchable — flip the scan signal *early* so
// UI progress polls see the picker as "ready" while we run the
// optional post-scan steps in the background.
signals.scanning.store(false, Ordering::Relaxed);
// in case we do a rescan, we have to resubscribe a watcher to the new set of directories
// all the already watched directories are not going to be resubscribed
if !config.install_watcher && !signals.cancelled.load(Ordering::Acquire) {
resubscribe_to_new_picker(&shared_picker);
}
// 3. Apply git status + frecency off-lock.
if !signals.cancelled.load(Ordering::Acquire)
&& let Some(status_handle) = status_handle
{
file_picker::apply_git_status_and_frecency(
&shared_picker,
&shared_frecency,
status_handle,
mode,
);
}
// 4. Install filesystem watcher (initial scan only).
if config.install_watcher && config.watch && !signals.cancelled.load(Ordering::Acquire) {
let shared_picker: &SharedFilePicker = &shared_picker;
let shared_frecency: &SharedFrecency = &shared_frecency;
let base_path: &std::path::Path = &base_path;
match BackgroundWatcher::new(
base_path.to_path_buf(),
git_workdir,
shared_picker.clone(),
shared_frecency.clone(),
mode,
) {
Ok(watcher) => {
if let Ok(mut guard) = shared_picker.write()
&& let Some(picker) = guard.as_mut()
{
picker.background_watcher = Some(watcher);
}
}
Err(e) => error!(?e, "failed to initialize background watcher"),
};
}
// 5. Post-scan warmup + bigram build.
if (config.warmup || config.content_indexing) && !signals.cancelled.load(Ordering::Acquire)
{
run_post_scan(&shared_picker, &base_path, &signals, &config);
}
// 6. Drain any rescan that arrived while we were busy.
//
// `trigger_full_rescan_async` sets `rescan_pending` whenever a
// caller asks for a rescan while `ScanJob::new` would have
// returned `Ok(None)` (scan active *or* post-scan busy). We
// consume the flag with `swap` so concurrent requests that land
// between the check and the follow-up spawn are still captured
// by the next invocation.
if !signals.cancelled.load(Ordering::Acquire)
&& signals.rescan_pending.swap(false, Ordering::AcqRel)
{
match Self::new(&shared_picker, &shared_frecency, false) {
Ok(Some(follow_up)) => {
info!("Rescheduling deferred rescan after current scan finished");
follow_up.spawn();
}
Ok(None) => {
// Another scan slipped in between our post-scan exit
// and the `new()` call above. That scan will drain
// the flag we just cleared — but we re-arm it so it
// does.
signals.rescan_pending.store(true, Ordering::Release);
}
Err(e) => {
error!(?e, "Failed to reschedule deferred rescan");
}
}
}
}
}
/// RAII helper that flips the `scanning` signal on construction and
/// resets it on drop (so early-returns can't leave it stuck on `true`).
/// Also drives the `watcher_ready` signal on the initial-scan path.
struct ScanningGuard<'a> {
signals: &'a ScanSignals,
release_watcher_ready_on_drop: bool,
}
impl<'a> ScanningGuard<'a> {
fn new(signals: &'a ScanSignals, release_watcher_ready_on_drop: bool) -> Self {
signals.scanning.store(true, Ordering::Relaxed);
Self {
signals,
release_watcher_ready_on_drop,
}
}
}
impl Drop for ScanningGuard<'_> {
fn drop(&mut self) {
self.signals.scanning.store(false, Ordering::Relaxed);
if self.release_watcher_ready_on_drop {
self.signals.watcher_ready.store(true, Ordering::Release);
}
}
}
fn run_post_scan(
shared_picker: &SharedFilePicker,
base_path: &std::path::Path,
signals: &ScanSignals,
config: &ScanConfig,
) {
let phase_start = std::time::Instant::now();
// Auto-scale the cache budget before we take the files snapshot —
// warmup needs the final budget.
if config.auto_cache_budget
&& !signals.cancelled.load(Ordering::Acquire)
&& let Ok(mut guard) = shared_picker.write()
&& let Some(picker) = guard.as_mut()
&& !picker.has_explicit_cache_budget()
{
let (files, _, _) = picker.sync_data_snapshot();
picker.set_cache_budget(ContentCacheBudget::new_for_repo(files.len()));
}
let Some((files, indexable_count, budget, arena, _busy_guard)) = shared_picker
.read()
.ok()
.and_then(|guard| guard.as_ref().map(|p| snapshot_sync_data(p, signals)))
else {
return;
};
if config.warmup && !signals.cancelled.load(Ordering::Acquire) {
let t = std::time::Instant::now();
warmup_mmaps(files, &budget, base_path, arena);
info!(
"Warmup completed in {:.2}s (cached {} files, {} bytes)",
t.elapsed().as_secs_f64(),
budget.cached_count.load(Ordering::Relaxed),
budget.cached_bytes.load(Ordering::Relaxed),
);
}
if config.content_indexing && !signals.cancelled.load(Ordering::Acquire) {
let indexable_files = &files[..indexable_count.min(files.len())];
let (index, content_binary) =
build_bigram_index(indexable_files, &budget, base_path, arena);
if let Ok(mut guard) = shared_picker.write()
&& let Some(picker) = guard.as_mut()
{
for &idx in &content_binary {
if let Some(file) = picker.get_file_mut(idx) {
file.set_binary(true);
}
}
picker.set_bigram_index(index, BigramOverlay::new(indexable_count));
}
}
info!(
"Post-scan phase total: {:.2}s (warmup={}, content_indexing={})",
phase_start.elapsed().as_secs_f64(),
config.warmup,
config.content_indexing,
);
}
struct PostScanBusyGuard<'a>(&'a AtomicBool);
impl Drop for PostScanBusyGuard<'_> {
fn drop(&mut self) {
self.0.store(false, Ordering::Release);
}
}
/// Re-registers all the directories at the watcher
#[tracing::instrument(skip_all)]
fn resubscribe_to_new_picker(shared_picker: &SharedFilePicker) {
let Ok(guard) = shared_picker.read() else {
return;
};
let Some(picker) = guard.as_ref() else {
return;
};
let Some(watcher) = picker.background_watcher.as_ref() else {
return;
};
// Base path first — this is the watch that delivers `Create(Folder)`
// events for brand-new top-level subdirs. On rescan paths this
// watch is still alive (the BackgroundWatcher survives rescans), so
// the call is idempotent. Including it explicitly protects against
// any future refactor that could drop the initial base-path watch.
watcher.request_watch_dir(picker.base_path().to_path_buf());
picker.for_each_dir(|dir: &std::path::Path| {
watcher.request_watch_dir(dir.to_path_buf());
std::ops::ControlFlow::Continue(())
});
}
/// Take a `'static`-lifetime snapshot of `sync_data` pinned by a
/// post-scan busy guard. Concurrent rescans short-circuit while the
/// returned guard is alive, so the raw slice can't be freed from under
/// the warmup + bigram build that consumes it.
fn snapshot_sync_data<'a>(
picker: &crate::file_picker::FilePicker,
signals: &'a ScanSignals,
) -> (
&'static [crate::types::FileItem],
usize,
Arc<ContentCacheBudget>,
crate::simd_path::ArenaPtr,
PostScanBusyGuard<'a>,
) {
signals.post_scan_busy.store(true, Ordering::Release);
let busy = PostScanBusyGuard(&signals.post_scan_busy);
let (files, indexable_count, arena) = picker.sync_data_snapshot();
let ptr = files.as_ptr();
let len = files.len();
let static_files: &'static [crate::types::FileItem] =
unsafe { std::slice::from_raw_parts(ptr, len) };
(
static_files,
indexable_count,
picker.cache_budget_arc(),
arena,
busy,
)
}
crates/fff-core/src/shared.rs
+133 -27
View File
@@ -1,55 +1,125 @@
use std::path::PathBuf;
use std::sync::{Arc, RwLock, RwLockReadGuard, RwLockWriteGuard};
use std::time::Duration;
use std::path::{Path, PathBuf};
use std::sync::{Arc, RwLock, RwLockReadGuard, RwLockWriteGuard, Weak};
use std::time::{Duration, Instant};
use crate::error::Error;
use crate::file_picker::FilePicker;
use crate::frecency::FrecencyTracker;
use crate::git::GitStatusCache;
use crate::query_tracker::QueryTracker;
use crate::scan::ScanJob;
/// Poll `.git/index.lock` until it disappears (git write completed), giving up
/// after [`GIT_LOCK_MAX_WAIT`]. Used by [`SharedPicker::refresh_git_status`]
/// to avoid reading a half-updated index when the watcher fires mid-`git add`.
///
/// The wait is bounded and cheap: the lock file is typically cleared within
/// a few milliseconds of the git command exiting.
fn wait_for_git_index_lock_release(git_root: &Path) {
const GIT_LOCK_POLL: Duration = Duration::from_millis(10);
const GIT_LOCK_MAX_WAIT: Duration = Duration::from_millis(500);
let lock = git_root.join(".git").join("index.lock");
// Fast path: no lock present.
if !lock.exists() {
return;
}
let deadline = Instant::now() + GIT_LOCK_MAX_WAIT;
while lock.exists() && Instant::now() < deadline {
std::thread::sleep(GIT_LOCK_POLL);
}
if lock.exists() {
tracing::warn!(
"Proceeding with git status refresh despite lingering \
.git/index.lock at {} — will retry once it clears",
lock.display()
);
}
}
/// Thread-safe shared handle to the [`FilePicker`] instance.
/// This accumulates only asynchronous non-blocking operations against the
/// file picker: creating, triggering various rescans and so on.
///
/// Uses `parking_lot::RwLock` which is reader-fair — new readers are not
/// blocked when a writer is waiting, preventing search query stalls during
/// background bigram builds or watcher writes.
/// For blocking access use internal picker via `.read()` or `.write()`
///
/// `Clone` gives a new handle to the same picker (Arc clone).
/// `Default` creates an empty handle suitable for `Lazy::new(SharedPicker::default)`.
/// ```ignore
/// let shared_picker = SharedFilePicker::default();
///
/// if let Some(picker) = shared_picker.read()?.as_ref() {
/// let files = picker.fuzzy_search(&query, options);
/// println!("Found {} files", files.len());
/// } else {
/// println!("Picker not initialized");
/// }
/// ```
#[derive(Clone, Default)]
pub struct SharedPicker(pub(crate) Arc<parking_lot::RwLock<Option<FilePicker>>>);
pub struct SharedFilePicker(pub(crate) Arc<SharedPickerInner>);
impl std::fmt::Debug for SharedPicker {
pub struct SharedPickerInner {
picker: parking_lot::RwLock<Option<FilePicker>>,
}
impl Default for SharedPickerInner {
fn default() -> Self {
Self {
picker: parking_lot::RwLock::new(None),
}
}
}
/// Non-owning handle to a [`SharedPicker`].
#[derive(Clone)]
pub(crate) struct WeakFilePicker(Weak<SharedPickerInner>);
impl WeakFilePicker {
/// Try to promote the weak handle back to a strong [`SharedPicker`].
///
/// Returns `None` once every strong `SharedPicker` clone has been
/// dropped. Callers should treat that as "the picker is being
/// torn down" and exit their current iteration cleanly.
pub(crate) fn upgrade(&self) -> Option<SharedFilePicker> {
self.0.upgrade().map(SharedFilePicker)
}
}
impl std::fmt::Debug for SharedFilePicker {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_tuple("SharedPicker").field(&"..").finish()
}
}
impl SharedPicker {
impl SharedFilePicker {
pub fn read(&self) -> Result<parking_lot::RwLockReadGuard<'_, Option<FilePicker>>, Error> {
Ok(self.0.read())
Ok(self.0.picker.read())
}
pub fn write(&self) -> Result<parking_lot::RwLockWriteGuard<'_, Option<FilePicker>>, Error> {
Ok(self.0.write())
Ok(self.0.picker.write())
}
/// Produce a non-owning handle to the same inner picker.
/// Use it if you don't need to block internal threads from dropping while owning this ref
pub(crate) fn weaken(&self) -> WeakFilePicker {
WeakFilePicker(Arc::downgrade(&self.0))
}
/// Return `true` if this is an instance of the picker that requires a complicated post-scan
/// indexing/cache warmup job. The indexing is not crazy but it takes time.
pub fn need_complex_rebuild(&self) -> bool {
let guard = self.0.read();
let guard = self.0.picker.read();
guard
.as_ref()
.is_some_and(|p| p.need_enable_mmap_cache() || p.need_enable_content_indexing())
.is_some_and(|p| p.has_mmap_cache() || p.has_content_indexing())
}
/// Block until the background filesystem scan finishes.
/// Returns `true` if scan completed, `false` on timeout.
pub fn wait_for_scan(&self, timeout: Duration) -> bool {
let signal = {
let guard = self.0.read();
let guard = self.0.picker.read();
match &*guard {
Some(picker) => picker.scan_signal(),
Some(picker) => Arc::clone(&picker.signals.scanning),
None => return true,
}
};
@@ -67,16 +137,16 @@ impl SharedPicker {
/// Block until the background file watcher is ready.
/// Returns `true` if watcher ready, `false` on timeout.
pub fn wait_for_watcher(&self, timeout: Duration) -> bool {
let signal = {
let guard = self.0.read();
let watch_ready_signal = {
let guard = self.0.picker.read();
match &*guard {
Some(picker) => picker.watcher_signal(),
Some(picker) => Arc::clone(&picker.signals.watcher_ready),
None => return true,
}
};
let start = std::time::Instant::now();
while !signal.load(std::sync::atomic::Ordering::Acquire) {
while !watch_ready_signal.load(std::sync::atomic::Ordering::Acquire) {
if start.elapsed() >= timeout {
return false;
}
@@ -85,9 +155,37 @@ impl SharedPicker {
true
}
/// Trigger a full filesystem rescan without blocking the caller.
/// Performs a safe async rescan. Guarantees only single active rescan per picker.
/// If many rescans requested the last one guaranteed to be finished.
pub fn trigger_full_rescan_async(&self, shared_frecency: &SharedFrecency) -> Result<(), Error> {
match ScanJob::new(self, shared_frecency, /*install_watcher=*/ false)? {
Some(job) => {
job.spawn();
}
None => {
// A scan is already in flight — mark a follow-up as
// needed. The running scan's `run()` drains this flag
// and reschedules itself.
if let Ok(guard) = self.read()
&& let Some(picker) = guard.as_ref()
{
picker
.scan_signals()
.rescan_pending
.store(true, std::sync::atomic::Ordering::Release);
tracing::info!(
"Full rescan requested while another scan is active — \
deferred via rescan_pending flag"
);
}
}
}
Ok(())
}
/// Refresh git statuses for all indexed files.
pub fn refresh_git_status(&self, shared_frecency: &SharedFrecency) -> Result<usize, Error> {
use git2::StatusOptions;
use tracing::debug;
let git_status = {
@@ -101,13 +199,21 @@ impl SharedPicker {
picker.git_root()
);
// Wait briefly for any in-progress git operation to release
// its `.git/index.lock`. libgit2 reads `.git/index` directly
// and does NOT coordinate with the filesystem lock; if a
// writer is mid-atomic-rename (lock file exists, new index
// not yet swapped in), we would observe stale status data.
// This matters most for the background watcher, which
// typically fires refresh in response to the very events
// produced by that in-flight git write.
if let Some(root) = picker.git_root() {
wait_for_git_index_lock_release(root);
}
GitStatusCache::read_git_status(
picker.git_root(),
StatusOptions::new()
.include_untracked(true)
.recurse_untracked_dirs(true)
.include_unmodified(true)
.exclude_submodules(true),
&mut crate::git::default_status_options(),
)
};
crates/fff-core/src/simd_path.rs
+4 -7
View File
@@ -317,15 +317,11 @@ impl ChunkedPathStoreBuilder {
pub fn add_file_immediate(&mut self, rel_path: &str, filename_offset: u16) -> ChunkedString {
let path_bytes = rel_path.as_bytes();
let byte_len = rel_path.len();
let n_chunks = chunks_needed(byte_len);
let mut indices = ChunkIndices::with_capacity(n_chunks);
let mut indices = ChunkIndices::with_capacity(chunks_needed(byte_len));
for i in 0..n_chunks {
let chunk_start = i * SIMD_CHUNK_BYTES;
let chunk_end = (chunk_start + SIMD_CHUNK_BYTES).min(byte_len);
for chunk in path_bytes.chunks(SIMD_CHUNK_BYTES) {
let mut chunk_bytes = [0u8; SIMD_CHUNK_BYTES];
chunk_bytes[..chunk_end - chunk_start]
.copy_from_slice(&path_bytes[chunk_start..chunk_end]);
chunk_bytes[..chunk.len()].copy_from_slice(chunk);
let arena_idx = match self.chunk_dedup.get(&chunk_bytes) {
Some(&idx) => idx,
@@ -336,6 +332,7 @@ impl ChunkedPathStoreBuilder {
idx
}
};
indices.push(arena_idx);
}
crates/fff-core/src/types.rs
+44 -6
View File
@@ -350,6 +350,9 @@ impl FileItem {
path.starts_with(prefix)
}
/// Write `base_path + '/' + relative_path` into `buf` and return it
/// as `&Path`. Takes a fixed-size array so the buffer can live on
/// the stack (no heap allocation, no bounds checks in the hot loop).
pub(crate) fn write_absolute_path<'a>(
&self,
arena: ArenaPtr,
@@ -359,7 +362,6 @@ impl FileItem {
let base = base_path.as_os_str().as_encoded_bytes();
let base_len = base.len();
buf[..base_len].copy_from_slice(base);
// Add separator if base doesn't end with one
let sep_len = if base_len > 0 && base[base_len - 1] != b'/' {
buf[base_len] = b'/';
1
@@ -367,14 +369,31 @@ impl FileItem {
0
};
let rel_start = base_len + sep_len;
let mut rel_buf = [0u8; PATH_BUF_SIZE];
let rel = self.path.read_to_buf(arena, &mut rel_buf);
let rel_bytes = rel.as_bytes();
buf[rel_start..rel_start + rel_bytes.len()].copy_from_slice(rel_bytes);
let total = rel_start + rel_bytes.len();
let rel = self.path.read_to_buf(arena, &mut buf[rel_start..]);
let total = rel_start + rel.len();
Path::new(unsafe { std::str::from_utf8_unchecked(&buf[..total]) })
}
/// Write the relative path into `buf` and NUL-terminate, returning
/// a `&CStr`. Fixed-size array so the buffer is stack-allocatable.
///
/// Paired with a parent-directory fd this eliminates the per-file
/// absolute-path memcpy: `openat(dir_fd, cstr.as_ptr(), O_RDONLY)`
/// resolves the name relative to `dir_fd`.
pub(crate) fn write_relative_cstr<'a>(
&self,
arena: ArenaPtr,
buf: &'a mut [u8; PATH_BUF_SIZE],
) -> &'a std::ffi::CStr {
// Reserve the last byte for the NUL terminator.
let rel = self.path.read_to_buf(arena, &mut buf[..PATH_BUF_SIZE - 1]);
let n = rel.len();
buf[n] = 0;
// SAFETY: `buf[..=n]` ends with the NUL we just wrote and
// filesystem paths never contain interior NULs.
unsafe { std::ffi::CStr::from_bytes_with_nul_unchecked(&buf[..=n]) }
}
#[inline]
pub fn total_frecency_score(&self) -> i32 {
self.access_frecency_score as i32 + self.modification_frecency_score as i32
@@ -439,6 +458,25 @@ impl FileItem {
self.content = OnceLock::new();
}
pub fn update_metadata(
&mut self,
budget: &ContentCacheBudget,
modified_secs: Option<u64>,
new_size: Option<u64>,
) {
if let Some(modified) = modified_secs
&& self.modified < modified
{
self.modified = modified;
}
self.invalidate_mmap(budget);
if let Some(size) = new_size {
self.size = size;
}
}
/// Get the cached file contents or lazily load and cache them.
///
/// Returns `None` if the file is too large, empty, can't be opened, **or
crates/fff-core/tests/bigram_overlay_coherence_test.rs
+35 -61
View File
@@ -21,7 +21,9 @@ use tempfile::TempDir;
use fff_search::file_picker::{FFFMode, FilePicker, FuzzySearchOptions};
use fff_search::grep::{GrepMode, GrepSearchOptions, parse_grep_query};
use fff_search::{FilePickerOptions, PaginationArgs, QueryParser, SharedFrecency, SharedPicker};
use fff_search::{
FilePickerOptions, PaginationArgs, QueryParser, SharedFilePicker, SharedFrecency,
};
/// Stress test: 200 base files, 3 rounds of edits + deletes. New files
/// are tracked but NOT verified via grep (see Group 3 for that bug).
@@ -502,10 +504,11 @@ fn bigram_overlay_coherence_mixed_tombstones_and_overflow() {
}
{
// assert
let guard = shared_picker.read().unwrap();
let picker = guard.as_ref().unwrap();
// Tombstones work: deleted tokens are gone.
// deleted tokens are gon
for token in &deleted_tokens {
assert_eq!(
grep_count(picker, token),
@@ -517,16 +520,16 @@ fn bigram_overlay_coherence_mixed_tombstones_and_overflow() {
// Surviving base files still findable.
for (name, token) in &repo_files[15..] {
assert!(
grep_count(picker, token) >= 1,
grep_count(picker, token) == 1,
"surviving {name} token should be findable"
);
}
// BUG: Overflow files not findable via grep.
// New tokens needs to be findable
for token in &new_tokens {
assert!(
grep_count(picker, token) >= 1,
"BUG: overflow token {token} should be findable but bigram skips overflow"
grep_count(picker, token) == 1,
"BUG: overflow token {token} should be findable"
);
}
}
@@ -754,18 +757,16 @@ fn bigram_overlay_coherence_rescan_after_git_commit() {
// Phase 2: Commit and rescan.
git_add_and_commit(base, "batch edit");
{
let mut guard = shared_picker.write().unwrap();
let picker = guard.as_mut().unwrap();
picker
.trigger_rescan(&shared_frecency)
.expect("trigger_rescan should succeed");
}
shared_picker
.trigger_full_rescan_async(&shared_frecency)
.expect("rescan should succeed");
// After trigger_rescan, sync_data is replaced (and bigram_index dropped
// with it). Wait for the synchronous scan to finish.
wait_for_scan(&shared_picker);
assert!(
shared_picker.wait_for_scan(Duration::from_secs(15)),
"Timed out waiting for scan to complete"
);
// Verify the file list is refreshed: all base_count + 5 files should
// be present as base files (not overflow, since they're committed).
@@ -890,18 +891,14 @@ fn bigram_overlay_coherence_full_lifecycle_seed_edit_commit_rescan_edit() {
// -- Phase 2: Commit and rescan --
git_add_and_commit(base, "phase 1 changes");
shared_picker
.trigger_full_rescan_async(&shared_frecency)
.expect("rescan should succeed");
{
let mut guard = shared_picker.write().unwrap();
let picker = guard.as_mut().unwrap();
picker
.trigger_rescan(&shared_frecency)
.expect("rescan should succeed");
}
// After rescan, bigram is dropped with old FileSync. Grep falls back
// to full search, which is correct.
wait_for_scan(&shared_picker);
assert!(
shared_picker.wait_for_scan(Duration::from_secs(15)),
"Timed out waiting for scan to complete"
);
// Phase1 tokens should still be findable (now in base index).
{
@@ -1191,14 +1188,13 @@ fn bigram_overlay_coherence_fuzzy_search_after_rescan() {
// Commit and rescan.
git_add_and_commit(base, "add grpc, remove web");
{
let mut guard = shared_picker.write().unwrap();
let picker = guard.as_mut().unwrap();
picker
.trigger_rescan(&shared_frecency)
.expect("rescan should succeed");
}
wait_for_scan(&shared_picker);
shared_picker
.trigger_full_rescan_async(&shared_frecency)
.expect("rescan should succeed");
assert!(
shared_picker.wait_for_scan(Duration::from_secs(15)),
"Timed out waiting for scan to complete"
);
// After rescan, fuzzy search should reflect the committed state.
{
@@ -1338,35 +1334,13 @@ fn grep_count(picker: &FilePicker, query: &str) -> usize {
fn grep_without_overlay_count(picker: &FilePicker, query: &str) -> usize {
let parsed = parse_grep_query(query);
picker
.grep_without_overlay(&parsed, &grep_opts())
.matches
.len()
picker.grep_original(&parsed, &grep_opts()).matches.len()
}
/// Wait for scanning to finish (no bigram requirement).
/// Use after `trigger_rescan` which replaces sync_data but does not
/// rebuild the bigram index.
fn wait_for_scan(shared_picker: &SharedPicker) {
let deadline = std::time::Instant::now() + Duration::from_secs(15);
loop {
std::thread::sleep(Duration::from_millis(50));
let ready = shared_picker
.read()
.ok()
.map(|guard| guard.as_ref().map_or(false, |p| !p.is_scan_active()))
.unwrap_or(false);
if ready {
break;
}
assert!(
std::time::Instant::now() < deadline,
"Timed out waiting for scan to complete"
);
}
}
fn wait_for_bigram(shared_picker: &SharedPicker) {
fn wait_for_bigram(shared_picker: &SharedFilePicker) {
let deadline = std::time::Instant::now() + Duration::from_secs(10);
loop {
std::thread::sleep(Duration::from_millis(50));
@@ -1389,7 +1363,7 @@ fn wait_for_bigram(shared_picker: &SharedPicker) {
}
}
fn stop_picker(shared_picker: &SharedPicker) {
fn stop_picker(shared_picker: &SharedFilePicker) {
if let Ok(mut guard) = shared_picker.write() {
if let Some(ref mut picker) = *guard {
picker.stop_background_monitor();
@@ -1426,8 +1400,8 @@ fn git_add_and_commit(dir: &Path, msg: &str) {
git_run(dir, &["commit", "-m", msg]);
}
fn make_picker(base: &Path) -> (SharedPicker, SharedFrecency) {
let shared_picker = SharedPicker::default();
fn make_picker(base: &Path) -> (SharedFilePicker, SharedFrecency) {
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
crates/fff-core/tests/bigram_overlay_integration.rs
+11 -7
View File
@@ -7,7 +7,7 @@ use tempfile::TempDir;
use fff_search::file_picker::{FFFMode, FilePicker};
use fff_search::grep::{GrepMode, GrepSearchOptions, parse_grep_query};
use fff_search::{FilePickerOptions, SharedFrecency, SharedPicker};
use fff_search::{FilePickerOptions, SharedFilePicker, SharedFrecency};
/// Create a temp directory with some initial files, run the full picker lifecycle,
/// then modify a file and verify grep finds the new content.
@@ -25,7 +25,7 @@ fn modified_file_findable_via_overlay() {
.unwrap();
fs::write(base.join("gamma.txt"), "yet another file\nmore lines\n").unwrap();
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
@@ -36,6 +36,7 @@ fn modified_file_findable_via_overlay() {
enable_mmap_cache: true,
enable_content_indexing: true,
mode: FFFMode::Neovim,
watch: false, // we drive events manually
..Default::default()
},
)
@@ -150,7 +151,7 @@ fn modified_file_findable_via_overlay() {
let picker = guard.as_ref().unwrap();
let parsed = parse_grep_query("UNIQUE_NEEDLE");
let opts = grep_opts();
let result = picker.grep_without_overlay(&parsed, &opts);
let result = picker.grep_original(&parsed, &opts);
assert_eq!(
result.matches.len(),
0,
@@ -175,7 +176,7 @@ fn deleted_file_excluded_via_overlay() {
fs::write(base.join("keep.txt"), "keep this content\n").unwrap();
fs::write(base.join("remove.txt"), "DELETEME_TOKEN is here\n").unwrap();
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
@@ -186,6 +187,7 @@ fn deleted_file_excluded_via_overlay() {
enable_mmap_cache: true,
enable_content_indexing: true,
mode: FFFMode::Neovim,
watch: false, // we drive events manually
..Default::default()
},
)
@@ -244,7 +246,7 @@ fn new_file_findable_after_add() {
fs::write(base.join("existing.txt"), "original content\n").unwrap();
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
@@ -255,6 +257,7 @@ fn new_file_findable_after_add() {
enable_mmap_cache: true,
enable_content_indexing: true,
mode: FFFMode::Neovim,
watch: false, // we drive events manually
..Default::default()
},
)
@@ -314,7 +317,7 @@ fn modified_file_findable_via_regex_overlay() {
)
.unwrap();
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
@@ -325,6 +328,7 @@ fn modified_file_findable_via_regex_overlay() {
enable_mmap_cache: true,
enable_content_indexing: true,
mode: FFFMode::Neovim,
watch: false, // we drive events manually
..Default::default()
},
)
@@ -397,7 +401,7 @@ fn grep_for<'a>(picker: &'a FilePicker, query: &str) -> fff_search::grep::GrepRe
picker.grep(&parsed, &grep_opts())
}
fn wait_for_bigram(shared_picker: &SharedPicker) {
fn wait_for_bigram(shared_picker: &SharedFilePicker) {
let deadline = std::time::Instant::now() + Duration::from_secs(30);
loop {
std::thread::sleep(Duration::from_millis(50));
crates/fff-core/tests/fuzz_file_operations.rs
+5 -3
View File
@@ -20,7 +20,9 @@ use rand::{RngCore, SeedableRng};
use fff_search::file_picker::{FFFMode, FilePicker, FuzzySearchOptions};
use fff_search::grep::{GrepMode, GrepSearchOptions, parse_grep_query};
use fff_search::{FilePickerOptions, PaginationArgs, QueryParser, SharedFrecency, SharedPicker};
use fff_search::{
FilePickerOptions, PaginationArgs, QueryParser, SharedFilePicker, SharedFrecency,
};
const DOMAINS: &[&str] = &[
r#"
@@ -308,7 +310,7 @@ fn fuzz_file_operations_stress() {
git_init_and_commit(base);
t_git += t0.elapsed();
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
FilePicker::new_with_shared_state(
shared_picker.clone(),
@@ -711,7 +713,7 @@ fn fuzzy_search_paths(picker: &FilePicker, query: &str) -> Vec<String> {
.collect()
}
fn wait_for_bigram(shared_picker: &SharedPicker) {
fn wait_for_bigram(shared_picker: &SharedFilePicker) {
let deadline = std::time::Instant::now() + Duration::from_secs(10);
loop {
std::thread::sleep(Duration::from_millis(50));
crates/fff-core/tests/fuzz_git_watcher_stress.rs
+1534
View File
File diff suppressed because it is too large Load Diff
crates/fff-core/tests/new_directory_watcher_test.rs
+7 -5
View File
@@ -24,7 +24,9 @@ use tempfile::TempDir;
use fff_search::file_picker::{FFFMode, FilePicker};
use fff_search::grep::{GrepMode, GrepSearchOptions, parse_grep_query};
use fff_search::{FilePickerOptions, PaginationArgs, QueryParser, SharedFrecency, SharedPicker};
use fff_search::{
FilePickerOptions, PaginationArgs, QueryParser, SharedFilePicker, SharedFrecency,
};
// ═══════════════════════════════════════════════════════════════════════
// Helpers
@@ -54,8 +56,8 @@ fn git_init_and_commit(dir: &Path) {
git_run(dir, &["commit", "-m", "initial"]);
}
fn make_watched_picker(base: &Path) -> (SharedPicker, SharedFrecency) {
let shared_picker = SharedPicker::default();
fn make_watched_picker(base: &Path) -> (SharedFilePicker, SharedFrecency) {
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::noop();
FilePicker::new_with_shared_state(
@@ -75,7 +77,7 @@ fn make_watched_picker(base: &Path) -> (SharedPicker, SharedFrecency) {
}
/// Wait for the initial scan + watcher to be fully ready.
fn wait_ready(shared_picker: &SharedPicker) {
fn wait_ready(shared_picker: &SharedFilePicker) {
assert!(
shared_picker.wait_for_scan(Duration::from_secs(10)),
"Timed out waiting for initial scan"
@@ -89,7 +91,7 @@ fn wait_ready(shared_picker: &SharedPicker) {
/// Poll the picker until `predicate` returns true or timeout expires.
/// Returns the elapsed duration if successful, panics on timeout.
fn poll_until(
shared_picker: &SharedPicker,
shared_picker: &SharedFilePicker,
timeout: Duration,
description: &str,
predicate: impl Fn(&FilePicker) -> bool,
crates/fff-core/tests/watcher_stop_under_lock.rs
+165
View File
@@ -0,0 +1,165 @@
//! Regression test: stopping the background watcher while the caller
//! holds the [`SharedFilePicker`] write lock must NOT deadlock.
//!
//! There are two lock-ordering hazards the watcher has to handle:
//!
//! 1. The debouncer's event thread calls our handler, which wants
//! `shared_picker.write()` to apply events. `stop()` used to
//! `join()` that thread under the caller's write guard.
//!
//! 2. The owner thread registers new-directory watches and injects
//! their existing files. Previously it held the debouncer mutex
//! across `shared_picker.write()`, while `stop()` takes the
//! debouncer mutex under the caller's write guard — inverse
//! lock orders, classic deadlock.
//!
//! macOS FSEvents is the reliable reproducer for (1) because fresh
//! `fs::write()` calls inside a just-watched temp dir queue events
//! faster than the debounce tick can drain them. Creating new
//! subdirectories exercises (2) via the owner thread's `watch_tx`.
use std::fs;
use std::sync::mpsc;
use std::time::Duration;
use tempfile::TempDir;
use fff_search::file_picker::{FFFMode, FilePicker};
use fff_search::{FilePickerOptions, SharedFilePicker, SharedFrecency};
/// Run `f` on a worker thread, require it to finish within `timeout`,
/// panic with `msg` otherwise. The caller gets to describe what the
/// worker is doing so a hung test produces an actionable message.
fn run_with_deadlock_guard(
msg: &'static str,
timeout: Duration,
f: impl FnOnce() + Send + 'static,
) {
let (done_tx, done_rx) = mpsc::channel::<()>();
let worker = std::thread::Builder::new()
.name("deadlock-guard-worker".into())
.spawn(move || {
f();
let _ = done_tx.send(());
})
.expect("spawn worker");
match done_rx.recv_timeout(timeout) {
Ok(()) => {}
Err(_) => panic!("{msg}"),
}
worker.join().expect("worker panicked");
}
fn make_watched_picker(base: &std::path::Path) -> (SharedFilePicker, SharedFrecency) {
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
shared_picker.clone(),
shared_frecency.clone(),
FilePickerOptions {
base_path: base.to_string_lossy().to_string(),
enable_mmap_cache: false,
enable_content_indexing: false,
mode: FFFMode::Neovim,
watch: true,
..Default::default()
},
)
.expect("Failed to create FilePicker");
assert!(
shared_picker.wait_for_scan(Duration::from_secs(10)),
"initial scan never completed"
);
assert!(
shared_picker.wait_for_watcher(Duration::from_secs(10)),
"watcher never installed"
);
(shared_picker, shared_frecency)
}
/// Hazard (1): debouncer event handler is waiting on `shared_picker.write()`
/// while the caller joins it from under the same guard.
#[test]
fn stop_background_monitor_under_write_lock_does_not_deadlock_file_events() {
let tmp = TempDir::new().unwrap();
let base = tmp.path().to_path_buf();
for i in 0..4 {
fs::write(base.join(format!("file_{i}.txt")), format!("seed {i}\n")).unwrap();
}
let (shared_picker, _shared_frecency) = make_watched_picker(&base);
// Produce enough filesystem churn that the debouncer has events
// queued and is likely mid-handler by the time we call stop.
for round in 0..8 {
for i in 0..4 {
let path = base.join(format!("file_{i}.txt"));
fs::write(&path, format!("edit {round}-{i}\n")).unwrap();
}
}
// Give the kernel time to deliver events into the debouncer queue
// (50 ms = default debouncer tick).
std::thread::sleep(Duration::from_millis(60));
let sp = shared_picker.clone();
run_with_deadlock_guard(
"stop_background_monitor() deadlocked under shared_picker.write() — \
the debouncer thread is likely waiting on the same write lock \
while we join it",
Duration::from_secs(5),
move || {
let mut guard = sp.write().expect("write lock");
if let Some(ref mut picker) = *guard {
picker.stop_background_monitor();
}
},
);
}
/// Hazard (2): owner thread holds the debouncer mutex while waiting
/// on `shared_picker.write()`, and `stop()` takes the debouncer mutex
/// under the caller's write guard.
#[test]
fn stop_background_monitor_under_write_lock_does_not_deadlock_new_dirs() {
let tmp = TempDir::new().unwrap();
let base = tmp.path().to_path_buf();
fs::write(base.join("seed.txt"), "seed\n").unwrap();
let (shared_picker, _shared_frecency) = make_watched_picker(&base);
// Create a burst of new subdirectories with files inside. On Linux
// the watcher event thread sends each new dir to `watch_tx`, and
// the owner thread processes them (taking the debouncer mutex +
// `shared_picker.write()`). On macOS the owner thread still runs
// `track_files_from_new_directories`, which takes the write lock.
for d in 0..8 {
let sub = base.join(format!("sub_{d}"));
fs::create_dir(&sub).unwrap();
for f in 0..4 {
fs::write(sub.join(format!("f_{f}.txt")), format!("{d}-{f}\n")).unwrap();
}
}
std::thread::sleep(Duration::from_millis(120));
let sp = shared_picker.clone();
run_with_deadlock_guard(
"stop_background_monitor() deadlocked under shared_picker.write() — \
the watcher owner thread is likely holding the debouncer mutex and \
waiting on the same write lock while we try to take the debouncer \
mutex to tear it down",
Duration::from_secs(5),
move || {
let mut guard = sp.write().expect("write lock");
if let Some(ref mut picker) = *guard {
picker.stop_background_monitor();
}
},
);
}
crates/fff-core/tests/watcher_thread_lifecycle_test.rs
+197
View File
@@ -0,0 +1,197 @@
#![cfg(target_os = "linux")]
use std::fs;
use std::path::PathBuf;
use std::time::{Duration, Instant};
use tempfile::TempDir;
use fff_search::file_picker::{FFFMode, FilePicker};
use fff_search::{FilePickerOptions, SharedFilePicker, SharedFrecency};
/// Thread comm names Linux exposes via `/proc/self/task/*/comm` are
/// capped at `TASK_COMM_LEN - 1 = 15` bytes. Our owner thread is named
/// `"fff-watcher-owner"` (17 bytes), so what actually appears in
/// `/proc` is the 15-byte truncation below.
const WATCHER_OWNER_THREAD_NAME: &str = "fff-watcher-own";
/// Walk `/proc/self/task/*/comm` and return how many live threads
/// carry `name` as their `comm`.
fn count_live_threads_named(name: &str) -> usize {
let Ok(dir) = fs::read_dir("/proc/self/task") else {
return 0;
};
let mut count = 0usize;
for entry in dir.flatten() {
let comm_path = entry.path().join("comm");
if let Ok(content) = fs::read_to_string(&comm_path) {
if content.trim_end() == name {
count += 1;
}
}
}
count
}
/// Poll until the thread count matches `expected` or we hit `timeout`.
fn wait_for_thread_count(name: &str, expected: usize, timeout: Duration) -> usize {
let deadline = Instant::now() + timeout;
loop {
let count = count_live_threads_named(name);
if count == expected {
return count;
}
if Instant::now() >= deadline {
return count;
}
std::thread::sleep(Duration::from_millis(25));
}
}
fn seed_repo(base: &std::path::Path) {
fs::create_dir_all(base.join("src")).unwrap();
fs::write(base.join("README.md"), "# seed\n").unwrap();
fs::write(base.join("src/main.rs"), "fn main() {}\n").unwrap();
fs::write(base.join("src/lib.rs"), "// lib\n").unwrap();
let _ = std::process::Command::new("git")
.args(["init", "-q", "-b", "main"])
.current_dir(base)
.output();
}
fn spawn_watched_picker(base: PathBuf) -> (SharedFilePicker, SharedFrecency) {
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
shared_picker.clone(),
shared_frecency.clone(),
FilePickerOptions {
base_path: base.to_string_lossy().to_string(),
enable_mmap_cache: false,
enable_content_indexing: false,
mode: FFFMode::Neovim,
watch: true,
..Default::default()
},
)
.expect("FilePicker::new_with_shared_state");
assert!(
shared_picker.wait_for_scan(Duration::from_secs(10)),
"initial scan did not complete"
);
assert!(
shared_picker.wait_for_watcher(Duration::from_secs(10)),
"watcher did not install"
);
(shared_picker, shared_frecency)
}
#[test]
fn watcher_threads_do_not_leak_across_picker_lifetimes() {
// this is needed because I run this within neovim with it's own fff owner thread lmao
let baseline = count_live_threads_named(WATCHER_OWNER_THREAD_NAME);
const PICKER_COUNT: usize = 4;
let mut tmpdirs: Vec<TempDir> = (0..PICKER_COUNT)
.map(|_| TempDir::new().expect("mktemp"))
.collect();
for td in &tmpdirs {
seed_repo(td.path());
}
let mut pickers: Vec<(SharedFilePicker, SharedFrecency)> = tmpdirs
.iter()
.map(|td| spawn_watched_picker(td.path().canonicalize().expect("canonicalize tmp")))
.collect();
let peak = wait_for_thread_count(
WATCHER_OWNER_THREAD_NAME,
baseline + PICKER_COUNT,
Duration::from_secs(5),
);
assert_eq!(
peak,
baseline + PICKER_COUNT,
"expected {} watcher-owner threads alive (baseline {} + {} pickers), saw {}",
baseline + PICKER_COUNT,
baseline,
PICKER_COUNT,
peak,
);
for i in 0..PICKER_COUNT {
let expected_remaining = baseline + PICKER_COUNT - (i + 1);
let (sp, sf) = pickers.remove(0);
drop(sp);
drop(sf);
let count = wait_for_thread_count(
WATCHER_OWNER_THREAD_NAME,
expected_remaining,
Duration::from_secs(5),
);
assert_eq!(
count,
expected_remaining,
"after dropping picker {}/{}: expected {} owner threads, saw {}",
i + 1,
PICKER_COUNT,
expected_remaining,
count,
);
}
tmpdirs.clear();
let after_stage1 = count_live_threads_named(WATCHER_OWNER_THREAD_NAME);
assert_eq!(
after_stage1, baseline,
"stage 1 leaked watcher-owner threads: baseline {}, observed {}",
baseline, after_stage1,
);
const ROUNDS: usize = 3;
for round in 0..ROUNDS {
let tmp = TempDir::new().expect("mktemp");
seed_repo(tmp.path());
let base = tmp.path().canonicalize().expect("canonicalize tmp");
let (sp, sf) = spawn_watched_picker(base);
let during = wait_for_thread_count(
WATCHER_OWNER_THREAD_NAME,
baseline + 1,
Duration::from_secs(5),
);
assert_eq!(
during,
baseline + 1,
"round {round}: expected 1 owner thread during run, saw {during} \
(baseline {baseline})",
);
drop(sp);
drop(sf);
drop(tmp);
let after =
wait_for_thread_count(WATCHER_OWNER_THREAD_NAME, baseline, Duration::from_secs(5));
assert_eq!(
after, baseline,
"round {round}: owner thread leaked after teardown \
(baseline {baseline}, observed {after})",
);
}
let final_count = count_live_threads_named(WATCHER_OWNER_THREAD_NAME);
assert_eq!(
final_count, baseline,
"watcher-owner threads leaked past the end of the test \
(baseline {}, final {})",
baseline, final_count,
);
}
crates/fff-mcp/src/main.rs
+2 -2
View File
@@ -15,7 +15,7 @@ mod update_check;
use clap::Parser;
use fff::file_picker::FilePicker;
use fff::frecency::FrecencyTracker;
use fff::{FFFMode, SharedFrecency, SharedPicker};
use fff::{FFFMode, SharedFilePicker, SharedFrecency};
use git2::Repository;
use mimalloc::MiMalloc;
use rmcp::{ServiceExt, transport::stdio};
@@ -232,7 +232,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
}
};
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
if let Some(frecency_db_path) = args.frecency_db_path {
match FrecencyTracker::new(&frecency_db_path, false) {
crates/fff-mcp/src/server.rs
+3 -3
View File
@@ -12,7 +12,7 @@ use crate::cursor::CursorStore;
use crate::output::{GrepFormatter, OutputMode, file_suffix};
use fff::grep::{GrepMode, GrepSearchOptions, has_regex_metacharacters};
use fff::types::{FileItem, PaginationArgs};
use fff::{FuzzySearchOptions, QueryParser, SharedFrecency, SharedPicker};
use fff::{FuzzySearchOptions, QueryParser, SharedFilePicker, SharedFrecency};
use fff_query_parser::AiGrepConfig;
use rmcp::handler::server::router::tool::ToolRouter;
use rmcp::handler::server::wrapper::Parameters;
@@ -160,7 +160,7 @@ pub struct MultiGrepParams {
#[derive(Clone)]
pub struct FffServer {
picker: SharedPicker,
picker: SharedFilePicker,
#[allow(dead_code)]
frecency: SharedFrecency,
cursor_store: Arc<Mutex<CursorStore>>,
@@ -169,7 +169,7 @@ pub struct FffServer {
}
impl FffServer {
pub fn new(picker: SharedPicker, frecency: SharedFrecency) -> Self {
pub fn new(picker: SharedFilePicker, frecency: SharedFrecency) -> Self {
Self {
picker,
frecency,
crates/fff-nvim/Cargo.toml
+3 -2
View File
@@ -77,8 +77,9 @@ ignore = "0.4.22"
mimalloc = { version = "0.1.47", features = ["local_dynamic_tls"] }
mlua = { version = "0.11.1", features = ["module", "luajit"] }
neo_frizbee = { workspace = true }
notify = "8.1.0"
notify-debouncer-full = "0.6"
notify = { workspace = true }
notify-debouncer-full = { workspace = true }
parking_lot = { workspace = true }
once_cell = "1.20.2"
pathdiff = "0.2.1"
serde = { version = "1.0", features = ["derive"] }
crates/fff-nvim/benches/grep_bench.rs
+6 -5
View File
@@ -1,13 +1,14 @@
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
use fff::file_picker::{FFFMode, FilePicker};
use fff::{
FilePickerOptions, GrepMode, GrepSearchOptions, SharedFrecency, SharedPicker, parse_grep_query,
FilePickerOptions, GrepMode, GrepSearchOptions, SharedFilePicker, SharedFrecency,
parse_grep_query,
};
use std::sync::OnceLock;
use std::time::Duration;
struct TestData {
shared_picker: SharedPicker,
shared_picker: SharedFilePicker,
}
static SETUP: OnceLock<TestData> = OnceLock::new();
@@ -32,7 +33,7 @@ fn big_repo_path() -> String {
fn setup() -> &'static TestData {
SETUP.get_or_init(|| {
let path = big_repo_path();
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
eprintln!("Initializing FilePicker for {:?}...", path);
@@ -70,9 +71,9 @@ fn setup() -> &'static TestData {
})
}
fn setup_cold() -> SharedPicker {
fn setup_cold() -> SharedFilePicker {
let path = big_repo_path();
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
FilePicker::new_with_shared_state(
crates/fff-nvim/benches/indexing_and_search.rs
+7 -7
View File
@@ -3,7 +3,7 @@ use fff::file_picker::{FFFMode, FilePicker};
use fff::types::PaginationArgs;
use fff::{
FilePickerOptions, FuzzySearchOptions, GrepMode, GrepSearchOptions, QueryParser,
SharedFrecency, SharedPicker,
SharedFilePicker, SharedFrecency,
};
use std::path::PathBuf;
use std::time::Duration;
@@ -25,7 +25,7 @@ fn init_tracing() {
/// Initialize FilePicker using shared state
fn init_file_picker_internal(
path: &str,
shared_picker: &SharedPicker,
shared_picker: &SharedFilePicker,
shared_frecency: &SharedFrecency,
) -> Result<(), String> {
FilePicker::new_with_shared_state(
@@ -43,7 +43,7 @@ fn init_file_picker_internal(
/// Helper function to wait for scanning to complete and get file count
fn wait_for_scan_completion(
shared_picker: &SharedPicker,
shared_picker: &SharedFilePicker,
timeout_secs: u64,
) -> Result<usize, String> {
let start = std::time::Instant::now();
@@ -105,7 +105,7 @@ fn wait_for_scan_completion(
}
/// Clean up shared state
fn cleanup_shared_state(shared_picker: &SharedPicker) {
fn cleanup_shared_state(shared_picker: &SharedFilePicker) {
if let Ok(mut picker_guard) = shared_picker.write() {
if let Some(mut picker) = picker_guard.take() {
picker.stop_background_monitor();
@@ -114,7 +114,7 @@ fn cleanup_shared_state(shared_picker: &SharedPicker) {
}
/// Initialize FilePicker once and return shared state
fn setup_once() -> Result<(SharedPicker, SharedFrecency), String> {
fn setup_once() -> Result<(SharedFilePicker, SharedFrecency), String> {
init_tracing();
let big_repo_path = PathBuf::from("./big-repo");
@@ -126,7 +126,7 @@ fn setup_once() -> Result<(SharedPicker, SharedFrecency), String> {
.map_err(|e| format!("Failed to canonicalize path: {}", e))?;
eprintln!(" Path: {:?}", canonical_path);
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
init_file_picker_internal(
@@ -171,7 +171,7 @@ fn bench_indexing(c: &mut Criterion) {
group.bench_function("index_big_repo", |b| {
b.iter(|| {
let sp = SharedPicker::default();
let sp = SharedFilePicker::default();
let sf = SharedFrecency::default();
let start = std::time::Instant::now();
crates/fff-nvim/src/bin/jemalloc_profile.rs
+3 -3
View File
@@ -1,5 +1,5 @@
use fff::file_picker::{FFFMode, FilePicker};
use fff::{FuzzySearchOptions, PaginationArgs, QueryParser, SharedFrecency, SharedPicker};
use fff::{FuzzySearchOptions, PaginationArgs, QueryParser, SharedFilePicker, SharedFrecency};
use std::env;
use std::thread;
use std::time::Duration;
@@ -60,7 +60,7 @@ fn format_bytes(bytes: usize) -> String {
}
fn test_search_memory_pattern(
shared_picker: &SharedPicker,
shared_picker: &SharedFilePicker,
name: &str,
iterations: usize,
query_pattern: impl Fn(usize) -> String,
@@ -177,7 +177,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
println!();
// Create shared state
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
// Initialize FilePicker
crates/fff-nvim/src/bin/search_profiler.rs
+3 -3
View File
@@ -1,9 +1,9 @@
use fff::file_picker::{FFFMode, FilePicker};
use fff::{FuzzySearchOptions, PaginationArgs, QueryParser, SharedFrecency, SharedPicker};
use fff::{FuzzySearchOptions, PaginationArgs, QueryParser, SharedFilePicker, SharedFrecency};
use std::time::{Duration, Instant};
/// Wait for background scan to complete
fn wait_for_scan(shared_picker: &SharedPicker, timeout_secs: u64) -> Result<usize, String> {
fn wait_for_scan(shared_picker: &SharedFilePicker, timeout_secs: u64) -> Result<usize, String> {
let timeout = Duration::from_secs(timeout_secs);
if !shared_picker.wait_for_scan(timeout) {
return Err(format!("Scan timed out after {} seconds", timeout_secs));
@@ -33,7 +33,7 @@ fn main() {
fff::path_utils::canonicalize(&big_repo_path).expect("Failed to canonicalize path");
// Create shared state
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
eprintln!("Initializing FilePicker for: {:?}", canonical_path);
crates/fff-nvim/src/bin/test_memory_leak.rs
+3 -10
View File
@@ -1,5 +1,5 @@
use fff::file_picker::{FFFMode, FilePicker};
use fff::{FuzzySearchOptions, PaginationArgs, QueryParser, SharedFrecency, SharedPicker};
use fff::{FuzzySearchOptions, PaginationArgs, QueryParser, SharedFilePicker, SharedFrecency};
use std::env;
use std::io::{self, Write};
use std::thread;
@@ -78,7 +78,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
println!();
// Create shared state
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
// Initialize the file picker
@@ -126,14 +126,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
// If async scan didn't work, trigger a manual scan
if !scan_completed {
println!("Triggering manual rescan...");
if let Ok(mut guard) = shared_picker.write()
&& let Some(ref mut picker) = *guard
{
match picker.trigger_rescan(&shared_frecency) {
Ok(_) => println!("Manual rescan completed"),
Err(e) => println!("Manual rescan failed: {:?}", e),
}
}
let _ = shared_picker.trigger_full_rescan_async(&shared_frecency);
}
let initial_file_count = {
crates/fff-nvim/src/bin/test_watcher.rs
+4 -2
View File
@@ -4,7 +4,9 @@
use fff::file_picker::FilePicker;
use fff::git::format_git_status;
use fff::{FFFMode, FuzzySearchOptions, PaginationArgs, QueryParser, SharedFrecency, SharedPicker};
use fff::{
FFFMode, FuzzySearchOptions, PaginationArgs, QueryParser, SharedFilePicker, SharedFrecency,
};
use std::env;
use std::io::{self, Write};
use std::sync::Arc;
@@ -25,7 +27,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
let r = running.clone();
// Create shared state
let shared_picker = SharedPicker::default();
let shared_picker = SharedFilePicker::default();
let shared_frecency = SharedFrecency::default();
// Clone for signal handler
crates/fff-nvim/src/lib.rs
+141 -107
View File
@@ -6,7 +6,7 @@ use fff::path_utils::expand_tilde;
use fff::query_tracker::QueryTracker;
use fff::{
DbHealthChecker, Error, FFFMode, FileSearchConfig, FuzzySearchOptions, GrepConfig,
PaginationArgs, QueryParser, Score, SearchResult, SharedFrecency, SharedPicker,
PaginationArgs, QueryParser, Score, SearchResult, SharedFilePicker, SharedFrecency,
SharedQueryTracker,
};
use mimalloc::MiMalloc;
@@ -27,7 +27,7 @@ static GLOBAL: MiMalloc = MiMalloc;
// the global state for neovim lives here for efficiency
// lua ffi is pretty bad with the overhead of converting raw pointer into tables
pub static FILE_PICKER: Lazy<SharedPicker> = Lazy::new(SharedPicker::default);
pub static FILE_PICKER: Lazy<SharedFilePicker> = Lazy::new(SharedFilePicker::default);
pub static FRECENCY: Lazy<SharedFrecency> = Lazy::new(SharedFrecency::default);
pub static QUERY_TRACKER: Lazy<SharedQueryTracker> = Lazy::new(SharedQueryTracker::default);
@@ -92,19 +92,22 @@ pub fn init_file_picker(_: &Lua, base_path: String) -> LuaResult<bool> {
}
fn reinit_file_picker_internal(path: &Path) -> Result<(), Error> {
// Cancel and stop the old picker under a single write lock to avoid
// a window where FILE_PICKER is None (which causes FilePickerMissing
// errors if the UI is searching concurrently).
// Cancel and stop the old picker's watcher under the write lock.
// `stop_background_monitor` is non-blocking (signals the debouncer
// to exit on its next tick without joining), so it's safe under
// the lock. In-flight watcher handlers finish naturally once we
// release the guard.
{
let mut guard = FILE_PICKER.write()?;
if let Some(ref mut picker) = *guard {
// Signal cancellation BEFORE stopping — this tells any orphaned
// scan threads from this picker to discard their results.
// Signal cancellation BEFORE stopping the watcher so any
// orphaned scan/post-scan threads discard their results
// instead of racing with the new picker.
picker.cancel();
picker.stop_background_monitor();
}
// Don't take() here — leave the old picker in place so searches
// still work until new_with_shared_state replaces it atomically.
// Don't take() the picker here — leave the old one in place so
// searches still work until new_with_shared_state replaces it.
}
// Create new picker — this atomically replaces the old one via write lock
@@ -136,14 +139,37 @@ pub fn restart_index_in_path(_: &Lua, new_path: String) -> LuaResult<()> {
LuaError::RuntimeError(format!("Failed to canonicalize path '{}': {}", new_path, e))
})?;
if let Ok(Some(picker)) = FILE_PICKER.read().as_deref()
&& picker.base_path() == canonical_path
{
return Ok(()); // same dir
}
// Spawn a background thread to avoid blocking Lua/UI thread
// Spawn a background thread BEFORE touching the picker lock. The
// same-dir short-circuit previously called `FILE_PICKER.read()` on
// the lua/UI thread, which blocks if a reindex writer is already in
// flight — on repeated DirChanged events (e.g. LSP root switching
// right after closing the picker) that could freeze the nvim main
// loop for the entire duration of an in-progress scan. Move the
// check into the spawned worker so the main thread always returns
// instantly; the internal reinit path also re-checks and no-ops if
// the picker is already pointing at `canonical_path`.
std::thread::spawn(move || {
::tracing::info!(
?canonical_path,
"restart_index_in_path: spawned worker running"
);
{
let guard = match FILE_PICKER.read() {
Ok(g) => g,
Err(_) => return,
};
if let Some(ref picker) = *guard
&& picker.base_path() == canonical_path
{
::tracing::info!(?canonical_path, "restart_index_in_path: same dir, skipping");
return;
}
}
::tracing::info!(
?canonical_path,
"restart_index_in_path: calling reinit_file_picker_internal"
);
if let Err(e) = reinit_file_picker_internal(&canonical_path) {
::tracing::error!(
?e,
@@ -159,14 +185,13 @@ pub fn restart_index_in_path(_: &Lua, new_path: String) -> LuaResult<()> {
}
pub fn scan_files(_: &Lua, _: ()) -> LuaResult<()> {
let mut file_picker = FILE_PICKER.write().into_lua_result()?;
let picker = file_picker
.as_mut()
.ok_or(Error::FilePickerMissing)
// Async: spawns a BG thread that walks off-lock, swaps sync_data
// under a brief write, then applies git + frecency off-lock. Returns
// immediately so the lua/UI thread never waits on the walk.
FILE_PICKER
.trigger_full_rescan_async(&FRECENCY)
.into_lua_result()?;
picker.trigger_rescan(&FRECENCY).into_lua_result()?;
::tracing::info!("scan_files trigger_rescan completed");
::tracing::info!("scan_files rescan spawned");
Ok(())
}
@@ -370,32 +395,65 @@ fn build_file_path_fallback(lua: &Lua, path: &Path, total_files: usize) -> LuaRe
}
pub fn track_access(_: &Lua, file_path: String) -> LuaResult<bool> {
// Called from the nvim main thread on every BufEnter via a libuv
// async chain. The body does an LMDB write (~100-200 ms) and takes
// `FILE_PICKER.write()`, which — if it has to queue behind any
// concurrent writer (post-scan install, background watcher event
// apply) — can stall the UI for multiple seconds. Hand the work off
// to a detached OS thread and return immediately so the main loop
// never waits on us.
//
// Losing a frecency update on shutdown is acceptable: frecency is
// a best-effort signal and the next BufEnter for the same file will
// produce an equivalent update.
let file_path = PathBuf::from(&file_path);
std::thread::spawn(move || {
{
let frecency_guard = match FRECENCY.read() {
Ok(g) => g,
Err(e) => {
::tracing::debug!(?e, "track_access: frecency read lock poisoned");
return;
}
};
let Some(ref frecency) = *frecency_guard else {
return;
};
if let Err(e) = frecency.track_access(file_path.as_path()) {
::tracing::debug!(?e, ?file_path, "track_access: frecency DB write failed");
return;
}
}
// Track access in frecency DB (expensive LMDB write, ~100-200ms)
// Do this WITHOUT holding FILE_PICKER lock to avoid blocking searches
let frecency_guard = FRECENCY.read().into_lua_result()?;
let Some(ref frecency) = *frecency_guard else {
return Ok(false);
};
frecency
.track_access(file_path.as_path())
.into_lua_result()?;
drop(frecency_guard);
let mut file_picker = match FILE_PICKER.write() {
Ok(g) => g,
Err(e) => {
::tracing::debug!(?e, "track_access: file picker write lock poisoned");
return;
}
};
let Some(ref mut picker) = *file_picker else {
return;
};
// Quick lock to update single file's frecency score in picker
let mut file_picker = FILE_PICKER.write().into_lua_result()?;
let Some(ref mut picker) = *file_picker else {
return Err(error::to_lua_error(Error::FilePickerMissing));
};
let frecency_guard = FRECENCY.read().into_lua_result()?;
let Some(ref frecency) = *frecency_guard else {
return Ok(false);
};
picker
.update_single_file_frecency(&file_path, frecency)
.into_lua_result()?;
let frecency_guard = match FRECENCY.read() {
Ok(g) => g,
Err(e) => {
::tracing::debug!(?e, "track_access: frecency read lock poisoned on update");
return;
}
};
let Some(ref frecency) = *frecency_guard else {
return;
};
if let Err(e) = picker.update_single_file_frecency(&file_path, frecency) {
::tracing::debug!(
?e,
?file_path,
"track_access: update_single_file_frecency failed"
);
}
});
Ok(true)
}
@@ -463,13 +521,14 @@ pub fn update_single_file_frecency(_: &Lua, file_path: String) -> LuaResult<bool
}
pub fn stop_background_monitor(_: &Lua, _: ()) -> LuaResult<bool> {
// `stop_background_monitor` is non-blocking — the debouncer /
// owner threads exit on their next iteration, so it is safe to
// call under the FILE_PICKER write lock.
let mut file_picker = FILE_PICKER.write().into_lua_result()?;
let Some(ref mut picker) = *file_picker else {
return Err(error::to_lua_error(Error::FilePickerMissing));
};
picker.stop_background_monitor();
Ok(true)
}
@@ -490,27 +549,30 @@ pub fn cancel_scan(_: &Lua, _: ()) -> LuaResult<bool> {
}
pub fn track_query_completion(_: &Lua, (query, file_path): (String, String)) -> LuaResult<bool> {
// Get the project path before spawning thread
let project_path = {
let file_picker = FILE_PICKER.read().into_lua_result()?;
let Some(ref picker) = *file_picker else {
return Ok(false);
};
picker.base_path().to_path_buf()
};
// Canonicalize the file path before spawning thread
let file_path = match fff::path_utils::canonicalize(&file_path) {
Ok(path) => path,
Err(e) => {
tracing::warn!(?file_path, error = ?e, "Failed to canonicalize file path for tracking");
return Ok(false);
}
};
// Spawn background thread to do the actual tracking (expensive DB write)
// Everything runs on a background thread — including the
// `FILE_PICKER.read()` used to fetch `project_path`. Doing that read
// on the main (UI) thread stalled nvim whenever a reindex writer was
// in flight: parking_lot's fair queue makes a read block behind a
// pending writer, so the main thread would freeze for the full
// duration of the scan.
let query_tracker = QUERY_TRACKER.clone();
std::thread::spawn(move || {
let project_path = match FILE_PICKER.read() {
Ok(guard) => match *guard {
Some(ref picker) => picker.base_path().to_path_buf(),
None => return,
},
Err(_) => return,
};
let file_path = match fff::path_utils::canonicalize(&file_path) {
Ok(path) => path,
Err(e) => {
tracing::warn!(?file_path, error = ?e, "Failed to canonicalize file path for tracking");
return;
}
};
if let Ok(mut guard) = query_tracker.write()
&& let Some(tracker) = guard.as_mut()
&& let Err(e) = tracker.track_query_completion(&query, &project_path, &file_path)
@@ -547,16 +609,18 @@ pub fn get_historical_query(_: &Lua, offset: usize) -> LuaResult<Option<String>>
}
pub fn track_grep_query(_: &Lua, query: String) -> LuaResult<bool> {
let project_path = {
let file_picker = FILE_PICKER.read().into_lua_result()?;
let Some(ref picker) = *file_picker else {
return Ok(false);
};
picker.base_path().to_path_buf()
};
// Move the `FILE_PICKER.read()` into the spawned worker too —
// see the note on `track_query_completion` above for why.
let query_tracker = QUERY_TRACKER.clone();
std::thread::spawn(move || {
let project_path = match FILE_PICKER.read() {
Ok(guard) => match *guard {
Some(ref picker) => picker.base_path().to_path_buf(),
None => return,
},
Err(_) => return,
};
if let Ok(mut guard) = query_tracker.write()
&& let Some(ref mut tracker) = *guard
&& let Err(e) = tracker.track_grep_query(&query, &project_path)
@@ -604,39 +668,9 @@ pub fn parse_grep_query(lua: &Lua, query: String) -> LuaResult<LuaTable> {
}
pub fn wait_for_initial_scan(_: &Lua, timeout_ms: Option<u64>) -> LuaResult<bool> {
// Extract the scan signal Arc WITHOUT holding the read lock, so the
// scan thread can acquire the write lock to store its results.
// Holding a read lock while polling would deadlock: the scan thread
// needs a write lock to finish, but can't acquire it while we hold the read lock.
let scan_signal = {
let file_picker = FILE_PICKER.read().into_lua_result()?;
let picker = file_picker
.as_ref()
.ok_or(Error::FilePickerMissing)
.into_lua_result()?;
picker.scan_signal()
}; // read lock released here
let scanned = FILE_PICKER.wait_for_scan(Duration::from_millis(timeout_ms.unwrap_or(500)));
let timeout_ms = timeout_ms.unwrap_or(500);
let timeout_duration = Duration::from_millis(timeout_ms);
let start_time = std::time::Instant::now();
let mut sleep_duration = Duration::from_millis(1);
while scan_signal.load(std::sync::atomic::Ordering::Relaxed) {
if start_time.elapsed() >= timeout_duration {
::tracing::warn!("wait_for_initial_scan timed out after {}ms", timeout_ms);
return Ok(false);
}
std::thread::sleep(sleep_duration);
sleep_duration = std::cmp::min(sleep_duration * 2, Duration::from_millis(50));
}
::tracing::debug!(
"wait_for_initial_scan completed in {:?}",
start_time.elapsed()
);
Ok(true)
Ok(scanned)
}
pub fn init_tracing(
doc/fff.nvim.txt
+1 -1
View File
@@ -1,5 +1,5 @@
*fff.nvim.txt*
For Neovim >= 0.10.0 Last change: 2026 April 30
For Neovim >= 0.10.0 Last change: 2026 May 02
==============================================================================
Table of Contents *fff.nvim-table-of-contents*
lua/fff/core.lua
+41 -13
View File
@@ -46,22 +46,50 @@ local function setup_global_autocmds(config)
vim.api.nvim_create_autocmd('DirChanged', {
group = group,
callback = function()
-- Window-local `:lcd` / `:tcd` are per-window — they don't change the
-- effective project root for the picker, so bail before touching
-- anything else.
if vim.v.event.scope == 'window' then return end
if not state.initialized then return end
local new_cwd = vim.v.event.cwd
if state.initialized and new_cwd and new_cwd ~= config.base_path then
vim.schedule(function()
-- Delay require to avoid circular dependency: core -> main -> picker_ui -> file_picker -> core
local ok, picker = pcall(require, 'fff.main')
if not ok then
vim.notify('FFF: Failed to load main module: ' .. tostring(picker), vim.log.levels.ERROR)
return
end
local change_ok, err = pcall(picker.change_indexing_directory, new_cwd)
if not change_ok then
vim.notify('FFF: Failed to change indexing directory: ' .. tostring(err), vim.log.levels.ERROR)
end
end)
if not new_cwd or new_cwd == '' then return end
-- Canonicalize both sides before comparing. `vim.v.event.cwd` is
-- whatever the caller passed to `:cd` (often unexpanded, sometimes
-- containing `~` or symlinks), while `config.base_path` is the form
-- the picker was last re-indexed against (post-`expand`). Without
-- resolving symlinks + ensuring an absolute path, trivially
-- equivalent paths compare as different (`/private/var/x` vs
-- `/var/x` on macOS, resolved-vs-unresolved symlinks from LSP root
-- detection, etc.) and every such mismatch schedules a 450k-file
-- reindex through the Rust side.
local function canonicalize(p)
if not p or p == '' then return p end
local abs = vim.fn.fnamemodify(vim.fn.expand(p), ':p')
-- `:p` leaves a trailing slash on directories — strip for
-- comparison stability.
abs = abs:gsub('/+$', '')
local ok, resolved = pcall(vim.fn.resolve, abs)
return (ok and resolved ~= '') and resolved or abs
end
local new_canonical = canonicalize(new_cwd)
local base_canonical = canonicalize(config.base_path)
if new_canonical == base_canonical then return end
vim.schedule(function()
-- Delay require to avoid circular dependency: core -> main -> picker_ui -> file_picker -> core
local ok, picker = pcall(require, 'fff.main')
if not ok then
vim.notify('FFF: Failed to load main module: ' .. tostring(picker), vim.log.levels.ERROR)
return
end
local change_ok, err = pcall(picker.change_indexing_directory, new_canonical)
if not change_ok then
vim.notify('FFF: Failed to change indexing directory: ' .. tostring(err), vim.log.levels.ERROR)
end
end)
end,
desc = 'Automatically sync FFF directory changes',
})