Files
colbymchenry--codegraph/__tests__/search-query-parser.test.ts
andreinknv 56f6b3b485 feat(search): field-qualified queries (kind:/lang:/path:/name:) + fuzzy typo fallback (#131)
* feat(search): field-qualified queries (kind:/lang:/path:/name:) + fuzzy typo fallback

Two UX improvements that turn a free-text search into something a
real user can drive precisely.

1) Field-qualified queries.

A new query parser (src/search/query-parser.ts) splits the raw query
into structured filters and a free-text remainder:

  kind:function name:auth path:src/api authenticate

becomes
  { kinds: ['function'], nameFilters: ['auth'],
    pathFilters: ['src/api'], text: 'authenticate' }

Filters compose with the SearchOptions arg (intersection). Unknown
prefixes pass through as plain text so `query "TODO:"` keeps working.
Quoted values (`path:"my dir"`) handle whitespace. When the user
specifies only filters with no text, the search uses a filter-only
candidate scan instead of bailing out.

Recognised today:
  kind:        any NodeKind value
  lang:        any Language value (alias: language:)
  path:        case-insensitive substring of file_path
  name:        case-insensitive substring of node.name

2) Fuzzy fallback.

When BOTH FTS and LIKE return nothing AND the text is at least 3
chars, the resolver scans the distinct-name set with a bounded
Damerau-Levenshtein-style edit distance (≤2 for ≥5 chars, ≤1 for
4-char queries, off for shorter). Bounded edit-distance early-exits
once the row min exceeds maxDist, so this stays O(distinct-names *
avg-name-length) with a very low constant.

Verified live against ollama/ollama@v0.22.0:
  query "kind:function auth"          → only function-kind hits
  query "lang:go path:server route"   → Go files under server/
  query "getUssr"   (typo)            → finds getUser, SetUser
  query "confg"     (typo)            → finds Config

Full test suite: 380 passed.

* fix(search): address reviewer findings — tokenizer mid-token quotes, fuzzy fan-out cap, larger filter-only over-fetch, unit tests

Five fixes from independent review:

- parseQuery tokenizer: quotes that appear MID-token (path:"my dir/
  file") were not being recognised — only quotes at the start of a
  token were treated as quoted spans. The fixture path:"my dir"
  parsed as ['path:"my', 'dir"'] instead of ['path:"my dir"'].
  Tokeniser is now a single state machine that scans into a token
  until whitespace OR a quote, and recognises quotes anywhere within
  the token (skips to the matching close quote).

- searchNodesFuzzy: cap the per-name follow-up SQL queries at
  Math.max(limit*2, 50) AFTER edit-distance filtering. Without
  this, a project with many similar names (getUser1, getUser2...)
  could fan out far beyond limit queries before the inner-loop
  break kicks in.

- searchAllByFilters (filter-only no-text path): bumped over-fetch
  multiplier from 2× to 5× so a selective post-filter (e.g.
  path:src/very/specific/file.ts) doesn't return fewer than limit
  results despite the DB having matches.

- 23 new unit tests in __tests__/search-query-parser.test.ts:
  parseQuery covers known-field filter, lang/language alias,
  multiple kind: ORs, quoted spans (incl. mid-token), URL
  passthrough, empty-value passthrough, unknown prefix passthrough,
  unknown value passthrough, all-filters-no-text, empty input,
  20k-char input. boundedEditDistance covers identity, single
  insertion/deletion/substitution, length-difference shortcut,
  empty inputs, case-sensitivity, early-exit correctness.

Full test suite: 853 passed (up from 830).

* refactor(search): derive parser kind/lang sets from types.ts as const

Convert NodeKind and Language to runtime-iterable as const arrays
(NODE_KINDS, LANGUAGES) so the query parser imports the canonical
list instead of duplicating it. Also fix the path: JSDoc to say
substring (matches the .includes() impl).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 20:35:49 -05:00

143 lines
4.9 KiB
TypeScript

/**
* Unit tests for the field-qualified query parser and bounded
* edit distance — the two algorithms behind `kind:`/`lang:`/`path:`/
* `name:` filtering and the fuzzy typo fallback.
*/
import { describe, it, expect } from 'vitest';
import { parseQuery, boundedEditDistance } from '../src/search/query-parser';
describe('parseQuery', () => {
it('returns plain text for a query with no field prefixes', () => {
const r = parseQuery('authenticate user');
expect(r.text).toBe('authenticate user');
expect(r.kinds).toEqual([]);
expect(r.languages).toEqual([]);
expect(r.pathFilters).toEqual([]);
expect(r.nameFilters).toEqual([]);
});
it('extracts kind: filter and removes it from text', () => {
const r = parseQuery('kind:function auth');
expect(r.kinds).toEqual(['function']);
expect(r.text).toBe('auth');
});
it('extracts lang: and language: as the same filter family', () => {
const a = parseQuery('lang:typescript foo');
const b = parseQuery('language:typescript foo');
expect(a.languages).toEqual(['typescript']);
expect(b.languages).toEqual(['typescript']);
});
it('handles multiple kind: filters as an OR set', () => {
const r = parseQuery('kind:function kind:method auth');
expect(r.kinds.sort()).toEqual(['function', 'method']);
});
it('extracts path: and name: as substring filters (kept verbatim)', () => {
const r = parseQuery('path:src/api name:Handler');
expect(r.pathFilters).toEqual(['src/api']);
expect(r.nameFilters).toEqual(['Handler']);
});
it('preserves quoted spans as a single token (whitespace in path:)', () => {
const r = parseQuery('path:"my dir/file" foo');
expect(r.pathFilters).toEqual(['my dir/file']);
expect(r.text).toBe('foo');
});
it('passes URL-like tokens through to text (does not match http: as a field)', () => {
const r = parseQuery('http://example.com');
expect(r.text).toBe('http://example.com');
expect(r.kinds).toEqual([]);
});
it('passes empty-value tokens through as text (kind: → "kind:")', () => {
const r = parseQuery('kind: foo');
expect(r.kinds).toEqual([]);
// The trailing-colon token comes back as plain text
expect(r.text.includes('kind:')).toBe(true);
});
it('passes unknown field prefixes through as text (TODO: keeps the colon)', () => {
const r = parseQuery('TODO: needs review');
expect(r.text).toBe('TODO: needs review');
expect(r.kinds).toEqual([]);
});
it('rejects unknown values for kind: (passes the whole token to text)', () => {
const r = parseQuery('kind:invalid foo');
// Invalid kind value falls back to text
expect(r.kinds).toEqual([]);
expect(r.text).toContain('kind:invalid');
});
it('handles all-filters-no-text query', () => {
const r = parseQuery('kind:function lang:typescript');
expect(r.kinds).toEqual(['function']);
expect(r.languages).toEqual(['typescript']);
expect(r.text).toBe('');
});
it('survives empty input', () => {
const r = parseQuery('');
expect(r.text).toBe('');
expect(r.kinds).toEqual([]);
});
it('survives a very long input (no allocation explosion)', () => {
const huge = 'foo '.repeat(5000); // 20k chars
const r = parseQuery(huge);
expect(r.text.length).toBeGreaterThan(0);
});
});
describe('boundedEditDistance', () => {
it('returns 0 for identical strings', () => {
expect(boundedEditDistance('user', 'user', 2)).toBe(0);
});
it('returns 1 for a single substitution', () => {
expect(boundedEditDistance('user', 'usar', 2)).toBe(1);
});
it('returns 1 for a single insertion', () => {
expect(boundedEditDistance('user', 'users', 2)).toBe(1);
});
it('returns 1 for a single deletion', () => {
expect(boundedEditDistance('users', 'user', 2)).toBe(1);
});
it('returns 2 for a transposition (two edits in basic Levenshtein)', () => {
// 'aple' vs 'palp' would be 2; pick a clearer pair.
// 'foo' vs 'fou': substitution + insertion = 2 if different lengths.
expect(boundedEditDistance('confg', 'configX', 2)).toBe(2);
});
it('returns maxDist+1 when distance clearly exceeds budget', () => {
expect(boundedEditDistance('foo', 'completely-different', 2)).toBe(3);
});
it('respects length-difference shortcut', () => {
// |len(a) - len(b)| > maxDist must immediately be over budget
expect(boundedEditDistance('a', 'aaaaaaa', 2)).toBe(3);
});
it('handles empty inputs', () => {
expect(boundedEditDistance('', '', 2)).toBe(0);
expect(boundedEditDistance('a', '', 2)).toBe(1);
expect(boundedEditDistance('', 'abc', 2)).toBe(3);
});
it('is case-sensitive — caller must lowercase if case-insensitive match wanted', () => {
expect(boundedEditDistance('Foo', 'foo', 2)).toBe(1);
});
it('early-exits when row min exceeds budget (correctness, not just perf)', () => {
// 'aaaaa' vs 'bbbbb': distance is 5, well over budget 2
expect(boundedEditDistance('aaaaa', 'bbbbb', 2)).toBe(3);
});
});