feat(kernel): R7b Dart walker — dart module, vendored-grammar-C d4d8f3e + wasm byte-copy vendor, dart default-routed (#1386)
R7b batch 4 #4 — the FINAL R7b language (docs/design/dart-kernel-port-checklist.md is the authoritative quirk list). The fourth vendored-grammar-C language, with a twist: production dart resolved its wasm from tree-sitter-wasms, whose dart dependency is an UNPINNED github:UserNobody14/tree-sitter-dart — a routine dependency update would have silently changed dart's grammar. This PR byte-copies the shipping 0.1.13 artifact into src/extraction/wasm/ (VENDORED_WASM_LANGS += dart) and compiles the same-commit (d4d8f3e337d8) parser.c/scanner.c in the kernel — table identity proven by the kernel-grammar-parity row. crates.io tree-sitter-dart is the nielsenko fork (different lineage) — rejected. The center of gravity is THE SIBLING-BODY DOUBLE-WALK, reproduced bug-for-bug: dart attaches every function/method body as a NEXT SIBLING of its signature, and the TS walkers consume each body TWICE — once via resolveBody (attributed to the function/method) and once via the enclosing generic walk (attributed to the file/class). Duplicate local-function nodes with the SAME id under different parents, duplicated calls/instantiates refs, and file/class-attributed fn-ref twins all emit in the exact observed interleave (a dedicated fixture pins the duplicate-id rows; the bloc kind-census spot-check pins the counts). Also preserved (probe-pinned): the extractBareCall selector matrix (the first callTypes=[] language — cascades completely invisible, `?.` encodes like `.`, the `ConfigT.load()` calls+references double emission with no callee-of-call skip, capitalized-chain `Foo.create().run` re-encode, const-object callee names); the constructor hooks (unnamed ctor skipped, named ctors/factories renamed to the CTOR name with the class as returnType, `@override (T) m()` record-misparse rescued by class-name validation); operator methods minting `method "<anonymous>"`; static_final_declaration constants via the visitNode hook while instance fields mint NOTHING; the prefixed-return-type prefix bug (`other.OtherClass f()` → returnType `other`); enum `with` mixins silent vs `implements` working; anonymous extensions named after the ON type; deferred imports invisible; named-argument callbacks NOT fn-ref-captured (the Flutter `onPressed:` idiom — future accuracy PR, TS-side first); `async*`/`sync*` NOT async; value-refs with the LIVE dart sibling-body pull and the `$X`-vs-`${X}` interpolation asymmetry; dartdoc kept in all three comment forms with the annotation-broken chain. Gates: parity sweeps first-run 0-diff on shelf/bloc/flutter — 5,815 clean files byte-parity, deferrals 10/21/1341 ≈ the survey's 10/21/~1340 (both-arm grammar reality: empty object patterns — the sealed-class idiom — and unnamed `library;` dominate; --max-deferral 0.3); full-init dumps byte-identical ×3 (shelf 7,959 / bloc 40,026 / flutter 1,855,319 dump lines); bloc per-kind node census identical across arms (the double-walk duplicate rows survive the store identically); kernel-dart-parity suite (7 fixtures + in-memory CRLF variants + double-walk duplicate-id pin + generated-file skip pin + two defer pins); full suite 2,688 green ×2 with CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += dart (20 langs — R7b COMPLETE). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
+2
-1
@@ -11,7 +11,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
### New Features
|
||||
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, C++, Rust, C#, Ruby, PHP, Swift, Kotlin, Scala, R, Lua, and Luau projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, protobuf-, tokio-, rust-analyzer-, jellyfin-, rails-, symfony-, swift-nio-, kotlinx.coroutines-, ggplot2-, Kong-, and Scala-3-compiler-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, C++, Rust, C#, Ruby, PHP, Swift, Kotlin, Scala, Dart, R, Lua, and Luau projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, protobuf-, tokio-, rust-analyzer-, jellyfin-, rails-, symfony-, swift-nio-, kotlinx.coroutines-, ggplot2-, Kong-, Scala-3-compiler-, and Flutter-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages.
|
||||
- Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
|
||||
- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
|
||||
@@ -32,6 +32,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- Ruby files also parse with an up-to-date grammar now (the previously bundled build dated from early 2024), which fixes a misparse of safe-navigation operator-method calls (`recv&.!= x`) that had recorded the wrong callee name.
|
||||
- PHP files also parse with an up-to-date grammar now (the previously bundled build dated from 2023): files using modern PHP features — enum constants, PHP 8.4 property hooks, parenthesis-free `new X()->method()` chaining — no longer hit parse errors or misparses that dropped or garbled their symbols, so codebases like Symfony and Laravel index substantially more accurately.
|
||||
- Swift files also parse with an up-to-date grammar now (the previously bundled build dated from 2023): swift-testing `#expect`, `#Preview`-style macros, the `package` access modifier, and typed `throws` no longer produce parse errors that dropped surrounding symbols — server-side Swift projects like Vapor see the biggest recovery.
|
||||
- Dart's grammar is now bundled with CodeGraph itself instead of being resolved from a third-party package whose Dart build floated on an unpinned upstream reference — a routine dependency update can no longer silently change how Dart files parse.
|
||||
- Searching or exploring by field names now finds the code that defines them. A query made of object keys or API field names (`profileInfo isTrialEligible quotaInfo billingMethod`) used to return unrelated results while the defining files never appeared, because three retrieval steps each dropped multi-word camelCase terms: an internal case-comparison bug, a match step that only considered classes (never functions or methods), and exploration seeding that required exact symbol-name matches. All three are fixed — `codegraph_explore` with a bag of field names now surfaces the controllers and services that assemble those fields. (#1196)
|
||||
- `codegraph.json`'s `includeIgnored` works again for the "folder of repos" layout: when one `.gitignore` rule covers a parent directory (`/repos/`) holding several embedded git repositories, opting in the individual repos (`"includeIgnored": ["repos/a/"]` — the exact spelling `codegraph init`'s own hint suggests) previously matched nothing and indexed zero files, looping the same suggestion back at you. Both spellings now work — name the parent directory to opt in everything under it, or name individual repos to opt in just those — and the hint no longer re-suggests repos that are already configured. (#1295)
|
||||
- Method calls on literals (`", ".join(...)` in Python, `"x".split(...)` in JavaScript, and the like) no longer produce call edges to unrelated project functions that happen to share the builtin's name — a codebase with a function called `join`, `get`, or `update` could show phantom callers from every string-builtin use. Additionally, a function nested inside another function is now only matched as a call target from inside its container, since it isn't reachable from anywhere else. Blast-radius and affected-test results get cleaner on Python and JavaScript codebases especially. (#1230)
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
/** Block dartdoc for blockDoc. */
|
||||
void blockDoc() {}
|
||||
|
||||
/// Line doc kept.
|
||||
// Plain comment also kept?
|
||||
void mixedDoc() {}
|
||||
|
||||
// Only plain.
|
||||
void plainDoc() {}
|
||||
|
||||
extension type Meters(double value) {
|
||||
double get km => value / 1000;
|
||||
void report() {
|
||||
print(km);
|
||||
}
|
||||
}
|
||||
|
||||
class Action {
|
||||
@override
|
||||
(int, String) reduce() => (1, 'a');
|
||||
void caller() {
|
||||
this.own();
|
||||
super.parent();
|
||||
generic<int>(5);
|
||||
final t = reduce();
|
||||
}
|
||||
}
|
||||
|
||||
enum Flag { on, off }
|
||||
|
||||
void flagUse(Flag f) {
|
||||
final v = Flag.on;
|
||||
switch (f) {
|
||||
case Flag.off:
|
||||
use(v);
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
final topFinal = seedValue();
|
||||
const topConst = 42;
|
||||
var topVar = 7;
|
||||
int topTyped = 8;
|
||||
final multiA = 1, multiB = 2;
|
||||
|
||||
int seedValue() => 9;
|
||||
|
||||
void hostFn() {
|
||||
void localFn(int n) {
|
||||
inner(n);
|
||||
}
|
||||
localFn(3);
|
||||
int localWithNew() {
|
||||
final w = new Holder(1);
|
||||
return w.n;
|
||||
}
|
||||
localWithNew();
|
||||
}
|
||||
|
||||
class Holder {
|
||||
final int n;
|
||||
final untypedInit = 5;
|
||||
static var mutable = 3;
|
||||
Holder(this.n);
|
||||
Holder.other() : n = compute();
|
||||
void useNew() {
|
||||
final h = new Holder(2);
|
||||
void methodLocal() {
|
||||
h.touch();
|
||||
}
|
||||
methodLocal();
|
||||
}
|
||||
int get computed => helperCall();
|
||||
}
|
||||
|
||||
void takeConst() {
|
||||
pad(const EdgeInsets.all(8.0));
|
||||
make(const Holder(3));
|
||||
}
|
||||
|
||||
Future<int> asyncFn() async => 1;
|
||||
Stream<int> genFn() async* {
|
||||
yield 1;
|
||||
}
|
||||
Iterable<int> syncGen() sync* {
|
||||
yield 2;
|
||||
}
|
||||
|
||||
/// Doc for annotated.
|
||||
@deprecated
|
||||
void annotated() {}
|
||||
|
||||
@Deprecated('use other')
|
||||
class OldClass {}
|
||||
|
||||
@pragma('vm:entry-point')
|
||||
void pragged() {}
|
||||
|
||||
class Cfg {
|
||||
static const retries = 3;
|
||||
}
|
||||
|
||||
void reader() {
|
||||
use(Cfg.retries);
|
||||
final msg = 'retry $topConst times ${topFinal}';
|
||||
print(msg);
|
||||
}
|
||||
|
||||
void shadower() {
|
||||
final topConst = 1;
|
||||
use(topConst);
|
||||
}
|
||||
|
||||
void refTaker() {
|
||||
register(seedValue);
|
||||
obj.cb = seedValue;
|
||||
final table = [seedValue, hostFn];
|
||||
final map = {'k': seedValue, 'j': notDefinedHere};
|
||||
reg2(cb: seedValue);
|
||||
final alias = seedValue;
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
void assigned() {}
|
||||
void listed() {}
|
||||
void mapped() {}
|
||||
void selfStore() {}
|
||||
class H {
|
||||
var cb;
|
||||
void wire(dynamic selfStore) {
|
||||
this.cb = assigned;
|
||||
cb = selfStore;
|
||||
}
|
||||
}
|
||||
final tableTop = [listed];
|
||||
final mapTop = {'k': mapped};
|
||||
void aliased() {}
|
||||
final aliasTop = aliased;
|
||||
class K {
|
||||
static final aliasStatic = aliased;
|
||||
}
|
||||
void onlyNamed() {}
|
||||
void otherFn() {}
|
||||
void positional() {}
|
||||
void taker() {
|
||||
reg(cb: onlyNamed);
|
||||
reg2(handler: otherFn, plain: 1);
|
||||
reg3(positional);
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
import 'dart:async';
|
||||
import 'package:foo/bar.dart' as bar show Baz hide Qux;
|
||||
import 'pkg.dart' deferred as lazy;
|
||||
export 'src/out.dart' show Pub;
|
||||
part 'part1.dart';
|
||||
|
||||
/// Doc line one.
|
||||
/// Doc line two.
|
||||
void topFn(int a, String b) {
|
||||
var local = 5;
|
||||
int typed = 6;
|
||||
final con = 7;
|
||||
int uninit;
|
||||
uninit = 8;
|
||||
local = a;
|
||||
helper(a);
|
||||
obj.method(a);
|
||||
Config.setting;
|
||||
Config.load();
|
||||
w?.render();
|
||||
list..add(1)..add(2);
|
||||
final w2 = Widget(1);
|
||||
final w3 = new Widget(2);
|
||||
const e = EdgeInsets.all(8.0);
|
||||
Foo.create().run();
|
||||
lower().chain();
|
||||
print('sum ${a + compute()} $local');
|
||||
}
|
||||
|
||||
// plain comment
|
||||
int get topGetter => 42;
|
||||
set topSetter(int v) {}
|
||||
|
||||
class Widget extends Base with Mix1, Mix2 implements Draw, Paint {
|
||||
static const int kMax = 10;
|
||||
static final shared = Widget(0);
|
||||
final int size;
|
||||
int count = 0;
|
||||
var loose;
|
||||
late String name;
|
||||
Widget(this.size);
|
||||
Widget.named(this.size) { init(); }
|
||||
factory Widget.create() => Widget(1);
|
||||
Widget._() : size = 0;
|
||||
@override
|
||||
void render(Canvas c) { c.draw(); }
|
||||
static Widget make() => Widget(3);
|
||||
int get area => size * size;
|
||||
set area(int v) { count = v; }
|
||||
Future<void> load() async { await fetch(); }
|
||||
operator +(Widget o) => Widget(size + o.size);
|
||||
}
|
||||
|
||||
mixin Mix1 on Base {
|
||||
void mixMethod() { helper(0); }
|
||||
}
|
||||
|
||||
extension WidgetExt on Widget {
|
||||
void extMethod() { render(null); }
|
||||
}
|
||||
|
||||
enum Color { red, green, blue }
|
||||
|
||||
enum Status with Mix1 implements Draw {
|
||||
ok(200),
|
||||
err(500);
|
||||
final int code;
|
||||
const Status(this.code);
|
||||
bool get good => code < 400;
|
||||
static Status parse(int c) => ok;
|
||||
}
|
||||
|
||||
typedef IntFn = int Function(int);
|
||||
typedef void OldStyle(int x);
|
||||
|
||||
abstract class Base {}
|
||||
class Draw {}
|
||||
@@ -0,0 +1,42 @@
|
||||
library my.lib;
|
||||
|
||||
part of 'other.dart';
|
||||
|
||||
void params({int? a, required Widget child, String note = 'x'}) {}
|
||||
void optional([int b = 0, Widget? w]) {}
|
||||
void fnTypedParam(void cb(int x), int Function(String) modern) {}
|
||||
num numRet(num n) => n;
|
||||
dynamic dynRet(dynamic d) => d;
|
||||
Object objRet(Object o) => o;
|
||||
double dblRet(double d) => d;
|
||||
|
||||
external void externalFn();
|
||||
|
||||
class Redir {
|
||||
Redir();
|
||||
Redir.a() : this();
|
||||
const factory Redir.b() = RedirImpl;
|
||||
}
|
||||
|
||||
void lambdas(List<int> xs) {
|
||||
xs.forEach((e) => use(e));
|
||||
final f = (int a) {
|
||||
helper(a);
|
||||
};
|
||||
f(1);
|
||||
xs.map((e) => e * 2).toList();
|
||||
}
|
||||
|
||||
void bodyTypes(Object x) {
|
||||
if (x is Widget) {
|
||||
use(x);
|
||||
}
|
||||
final y = x as Widget;
|
||||
final list = <Widget>[];
|
||||
final map = <String, Widget>{};
|
||||
throw StateError('bad');
|
||||
}
|
||||
|
||||
@override
|
||||
@protected
|
||||
void doubleAnno() {}
|
||||
@@ -0,0 +1,61 @@
|
||||
import 'package:foo/util.dart' as util;
|
||||
|
||||
const SHARED_MAX = 10;
|
||||
const kLimit = 20;
|
||||
final DERIVED = SHARED_MAX + 1;
|
||||
const low = 30;
|
||||
|
||||
class Table {
|
||||
static const COL_LIMIT = 5;
|
||||
static const plain = 6;
|
||||
void reads() {
|
||||
use(SHARED_MAX);
|
||||
use(Table.COL_LIMIT);
|
||||
log('cap $kLimit and ${SHARED_MAX}');
|
||||
}
|
||||
void shadowed() {
|
||||
final SHARED_MAX = 1;
|
||||
use(SHARED_MAX);
|
||||
}
|
||||
}
|
||||
|
||||
void freeReader() {
|
||||
use(kLimit);
|
||||
}
|
||||
|
||||
void uninitShadow() {
|
||||
int DERIVED;
|
||||
DERIVED = 2;
|
||||
use(DERIVED);
|
||||
}
|
||||
|
||||
void assignOnly() {
|
||||
low = 5;
|
||||
}
|
||||
|
||||
void prefixedCalls() {
|
||||
util.helper(1);
|
||||
util.Config.load();
|
||||
Widget.named(2);
|
||||
}
|
||||
|
||||
/// Doc on classy.
|
||||
class Classy {}
|
||||
|
||||
/// Doc on num const?
|
||||
const DOCED = 1;
|
||||
|
||||
/// Doc on enum.
|
||||
enum E2 { a }
|
||||
|
||||
/// Doc broken.
|
||||
@override
|
||||
void afterAnno() {}
|
||||
|
||||
/// Kept doc.
|
||||
void unicodeNext() {}
|
||||
|
||||
// π and émoji 🎯 in a comment
|
||||
void afterUnicode(String s) {
|
||||
emit('café ☕ done');
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
/// File-level doc for torture (glued to import? no — imports precede nothing).
|
||||
import 'dart:async';
|
||||
import 'package:torture/other.dart' as other show OtherClass;
|
||||
export 'src/reexported.dart' hide Hidden;
|
||||
part 'torture_part.dart';
|
||||
|
||||
/// Doc line one.
|
||||
/// Doc line two.
|
||||
void topLevel(int count, String label) {
|
||||
helper(count);
|
||||
}
|
||||
|
||||
/** Block dartdoc kept. */
|
||||
int blockDoc() => 1;
|
||||
|
||||
// Plain comment doc.
|
||||
String plainDoc() => 'x';
|
||||
|
||||
/// Broken by annotation.
|
||||
@deprecated
|
||||
void annotated() {}
|
||||
|
||||
@Deprecated('with args')
|
||||
@pragma('vm:entry-point')
|
||||
void doubleAnno() {}
|
||||
|
||||
const SHARED_MAX = 10;
|
||||
final DERIVED_VAL = SHARED_MAX + 1;
|
||||
const lowercase_const = 1;
|
||||
final typedTop = compute();
|
||||
var topVar = 5;
|
||||
int topTyped = 6;
|
||||
final multiA = 1, multiB = 2;
|
||||
|
||||
int get topGetter => 7;
|
||||
set topSetter(int v) {}
|
||||
|
||||
Future<String> asyncFn() async => 'a';
|
||||
Stream<int> genStar() async* {
|
||||
yield 1;
|
||||
}
|
||||
Iterable<int> syncStar() sync* {
|
||||
yield 2;
|
||||
}
|
||||
|
||||
num numback(num n) => n;
|
||||
dynamic dyn(dynamic d) => d;
|
||||
Object obj(Object o) => o;
|
||||
List<WidgetT> listRet(Map<String, WidgetT> m) => [];
|
||||
WidgetT? nullableRet() => null;
|
||||
other.OtherClass prefixedRet() => other.OtherClass();
|
||||
T generic<T>(T v) => v;
|
||||
|
||||
external void externalFn();
|
||||
|
||||
void params({int? named, required WidgetT child, String note = 'x'}) {}
|
||||
void optionals([int pos = 0, WidgetT? w]) {}
|
||||
void fnTyped(void cb(int x), int Function(String) modern) {}
|
||||
|
||||
void bodyShapes(List<int> xs, Object o) {
|
||||
var local = 1;
|
||||
int typed = 2;
|
||||
final con = 3;
|
||||
int uninit;
|
||||
uninit = 4;
|
||||
local = uninit;
|
||||
helper(local);
|
||||
obj.method(local);
|
||||
this_like.deep.call3(local);
|
||||
ConfigT.load();
|
||||
ConfigT.setting;
|
||||
w?.render();
|
||||
y2..add(1)..add(2);
|
||||
final w1 = WidgetT(1);
|
||||
final w2 = new WidgetT(2);
|
||||
pad(const EdgeInsetsT.all(8.0));
|
||||
FactoryT.create().run();
|
||||
lower().chain();
|
||||
WidgetT.named(3).chainTail();
|
||||
xs.map((e) => e * 2).toList();
|
||||
xs.forEach((e) => use(e));
|
||||
final lam = (int a) {
|
||||
helper(a);
|
||||
};
|
||||
lam(5);
|
||||
void localFn(int n) {
|
||||
inner(n);
|
||||
}
|
||||
localFn(6);
|
||||
if (o is WidgetT) {
|
||||
use(o);
|
||||
}
|
||||
final cast = o as WidgetT;
|
||||
final tl = <WidgetT>[];
|
||||
throw StateError('bad');
|
||||
}
|
||||
|
||||
void interpolation(int count) {
|
||||
log('count $count and ${SHARED_MAX} via ${refresh()}');
|
||||
}
|
||||
|
||||
void refTaker() {
|
||||
register(topLevel);
|
||||
obj.cb = topLevel;
|
||||
final table = [topLevel, blockDoc];
|
||||
final m = {'k': topLevel, 'x': undefinedName};
|
||||
reg2(cb: topLevel);
|
||||
forward(topLevel: topLevel);
|
||||
final alias = topLevel;
|
||||
}
|
||||
|
||||
/// Class doc.
|
||||
@immutable
|
||||
class WidgetT extends BaseT with MixA, MixB implements DrawT {
|
||||
static const int K_MAX = 9;
|
||||
static final sharedInst = WidgetT(0);
|
||||
static var mutableStatic = 1;
|
||||
final int size;
|
||||
final untyped = 5;
|
||||
int counter = 0;
|
||||
late String title;
|
||||
WidgetT(this.size);
|
||||
WidgetT.named(this.size) {
|
||||
init();
|
||||
}
|
||||
WidgetT.bodilessNamed() : size = seed();
|
||||
factory WidgetT.create() => WidgetT(1);
|
||||
const factory WidgetT.redir() = WidgetT2;
|
||||
@override
|
||||
void render(CanvasT c) {
|
||||
c.draw();
|
||||
}
|
||||
static WidgetT make() => WidgetT(3);
|
||||
int get area => size * size;
|
||||
set area(int v) {
|
||||
counter = v;
|
||||
}
|
||||
bool get privado => _check();
|
||||
bool _check() => true;
|
||||
Future<void> load() async {
|
||||
await fetch();
|
||||
}
|
||||
WidgetT operator +(WidgetT o) => WidgetT(size + o.size);
|
||||
void useLocalNew() {
|
||||
final h = new HolderT(2);
|
||||
use(h);
|
||||
}
|
||||
void readsConst() {
|
||||
use(K_MAX);
|
||||
}
|
||||
}
|
||||
|
||||
class WidgetT2 extends WidgetT {
|
||||
WidgetT2() : super(0);
|
||||
void _privateMethod() {}
|
||||
}
|
||||
|
||||
class BaseT {}
|
||||
|
||||
class DrawT {}
|
||||
|
||||
class OnlyMix with MixA {}
|
||||
|
||||
abstract class AbstractT {
|
||||
void mustImpl(WidgetT w);
|
||||
int get abstractGetter;
|
||||
}
|
||||
|
||||
sealed class ShapeT {}
|
||||
|
||||
class CircleT extends ShapeT {}
|
||||
|
||||
mixin MixA on BaseT {
|
||||
void mixMethod() {
|
||||
helper(1);
|
||||
}
|
||||
}
|
||||
|
||||
mixin MixB implements DrawT {
|
||||
int get mixGetter => 2;
|
||||
}
|
||||
|
||||
extension WidgetTExt on WidgetT {
|
||||
void extMethod() {
|
||||
render(CanvasT());
|
||||
}
|
||||
|
||||
int get extGetter => 4;
|
||||
}
|
||||
|
||||
extension on String {
|
||||
void anonExt() {}
|
||||
}
|
||||
|
||||
enum ColorT { red, green, blue }
|
||||
|
||||
/// Enum doc.
|
||||
enum StatusT with MixB implements DrawT {
|
||||
ok(200),
|
||||
err(500);
|
||||
|
||||
final int code;
|
||||
const StatusT(this.code);
|
||||
bool get good => code < 400;
|
||||
static StatusT parse(int c) => ok;
|
||||
}
|
||||
|
||||
typedef IntFn = int Function(int);
|
||||
typedef void LegacyCb(int x);
|
||||
typedef MapAlias = Map<String, WidgetT>;
|
||||
|
||||
extension type MetersT(double value) {
|
||||
double get km => value / 1000;
|
||||
}
|
||||
|
||||
void patternUser(Object o) {
|
||||
final v = ColorT.red;
|
||||
switch (o) {
|
||||
case ColorT.blue:
|
||||
use(v);
|
||||
default:
|
||||
break;
|
||||
}
|
||||
final (a, b) = (1, 2);
|
||||
use(a);
|
||||
}
|
||||
|
||||
// π unicode: “smart quotes” précède — column check ☕
|
||||
void afterUnicode(String seance) {
|
||||
emit('café ☕ done');
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
/**
|
||||
* Kernel↔wasm Dart extraction parity (R7b batch 4 of the kernel migration —
|
||||
* the final R7b language).
|
||||
*
|
||||
* Asserts the native walker (codegraph-kernel/src/dart.rs) produces the SAME
|
||||
* ExtractionResult as the wasm TreeSitterExtractor — nodes, edges, and
|
||||
* unresolved refs compared as canonicalized multisets — over the checked-in
|
||||
* fixtures (torture.dart: the master inventory — imports incl. deferred
|
||||
* invisibility, dartdoc in all three comment forms with the
|
||||
* annotation-broken chain, stacked annotations in reverse order, the
|
||||
* static_final_declaration constants hook, the full ctor set with the
|
||||
* unnamed-ctor skip and named-ctor renaming, operator methods as
|
||||
* `<anonymous>`, the extractBareCall matrix incl. cascade invisibility and
|
||||
* `?.`-as-`.`, the `ConfigT.load()` calls+references double emission,
|
||||
* extends/with/implements ref kinds, enum `with` silence, anonymous
|
||||
* extensions named after the ON type, value-ref targets with the sibling
|
||||
* body pull; TortureDoubleWalk.dart: THE SIBLING-BODY DOUBLE-WALK — the
|
||||
* duplicate local-function nodes with the same id under different parents
|
||||
* and the exact duplicated-ref interleave; TortureFnrefDart.dart: fn-ref
|
||||
* capture channels incl. named-argument non-capture and the file/class
|
||||
* twins; TortureMini/TortureSigs/TortureCtors/TortureVrefDart: signatures
|
||||
* verbatim, prefixed-return-type prefix bug, const factories invisible,
|
||||
* value-ref matrix with `$X` vs `${X}` asymmetry) and their CRLF variants
|
||||
* (derived in-memory — #1329), plus defer and generated-file pins.
|
||||
*
|
||||
* The full-repo sweeps live in scripts/kernel-parity.mjs (shelf/bloc/flutter
|
||||
* with --max-deferral 0.3); this suite keeps the invariant alive in
|
||||
* `npm test`. Skips when no kernel binary is staged; CODEGRAPH_KERNEL_EXPECT=1
|
||||
* turns that into a failure.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { extractFromSource } from '../src/extraction';
|
||||
import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars';
|
||||
import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel';
|
||||
import type { ExtractionResult } from '../src/types';
|
||||
|
||||
const KERNEL_PATH = path.join(
|
||||
__dirname,
|
||||
'..',
|
||||
'codegraph-kernel',
|
||||
'prebuilds',
|
||||
`${process.platform}-${process.arch}`,
|
||||
'codegraph-kernel.node'
|
||||
);
|
||||
const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
const FIXTURE_DIR = path.join(__dirname, 'fixtures', 'kernel-parity');
|
||||
|
||||
function canon(result: ExtractionResult): { nodes: string[]; edges: string[]; refs: string[] } {
|
||||
return {
|
||||
nodes: result.nodes
|
||||
.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort()))
|
||||
.sort(),
|
||||
edges: result.edges.map((e) => JSON.stringify(e, Object.keys(e).sort())).sort(),
|
||||
refs: result.unresolvedReferences
|
||||
.map((r) => JSON.stringify(r, Object.keys(r).sort()))
|
||||
.sort(),
|
||||
};
|
||||
}
|
||||
|
||||
const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const;
|
||||
let savedEnv: Record<string, string | undefined>;
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel Dart extraction parity', () => {
|
||||
beforeAll(async () => {
|
||||
await initGrammars();
|
||||
await loadGrammarsForLanguages(['dart']);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]]));
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
for (const k of ENV_KEYS) {
|
||||
if (savedEnv[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = savedEnv[k];
|
||||
}
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
function assertParity(filePath: string, source: string, minNodes = 2): ExtractionResult {
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
const viaKernel = tryKernelExtract(filePath, source, 'dart');
|
||||
expect(viaKernel, `kernel extraction failed for ${filePath}`).not.toBeNull();
|
||||
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource(filePath, source, 'dart');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
|
||||
const k = canon(viaKernel!);
|
||||
const w = canon(viaWasm);
|
||||
expect(k.nodes, `${filePath}: nodes`).toEqual(w.nodes);
|
||||
expect(k.edges, `${filePath}: edges`).toEqual(w.edges);
|
||||
expect(k.refs, `${filePath}: refs`).toEqual(w.refs);
|
||||
expect(viaWasm.nodes.length).toBeGreaterThanOrEqual(minNodes);
|
||||
return viaKernel!;
|
||||
}
|
||||
|
||||
const FIXTURES = [
|
||||
['torture.dart', 30],
|
||||
['TortureDoubleWalk.dart', 5],
|
||||
['TortureFnrefDart.dart', 3],
|
||||
['TortureMini.dart', 5],
|
||||
['TortureSigs.dart', 4],
|
||||
['TortureCtors.dart', 3],
|
||||
['TortureVrefDart.dart', 4],
|
||||
] as const;
|
||||
|
||||
for (const [file, minNodes] of FIXTURES) {
|
||||
it(`${file}: parity`, () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, file), 'utf8');
|
||||
assertParity(`fixtures/${file}`, src, minNodes);
|
||||
});
|
||||
|
||||
it(`${file}: CRLF parity`, () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, file), 'utf8');
|
||||
const crlf = src.replace(/(?<!\r)\n/g, '\r\n');
|
||||
assertParity(`fixtures/${file} (crlf)`, crlf, minNodes);
|
||||
});
|
||||
}
|
||||
|
||||
it('double-walk pins: duplicate local-fn nodes share an id; refs interleave', () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, 'TortureDoubleWalk.dart'), 'utf8');
|
||||
const result = assertParity('fixtures/TortureDoubleWalk.dart', src, 5);
|
||||
// Local functions are minted TWICE — same (kind,name,line) → the SAME id
|
||||
// — once under the enclosing function, once under the file/class (the
|
||||
// sibling-body revisit). A dedupe here would silently diverge.
|
||||
const byId = new Map<string, number>();
|
||||
for (const n of result.nodes) byId.set(n.id, (byId.get(n.id) ?? 0) + 1);
|
||||
const dupes = [...byId.values()].filter((c) => c > 1);
|
||||
expect(dupes.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('generated files extract but skip fn-ref and value-ref flushes', () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, 'TortureVrefDart.dart'), 'utf8');
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
const viaKernel = tryKernelExtract('lib/model.g.dart', src, 'dart');
|
||||
expect(viaKernel).not.toBeNull();
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource('lib/model.g.dart', src, 'dart');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
const k = canon(viaKernel!);
|
||||
const w = canon(viaWasm);
|
||||
expect(k.nodes).toEqual(w.nodes);
|
||||
expect(k.edges).toEqual(w.edges);
|
||||
expect(k.refs).toEqual(w.refs);
|
||||
// The skips: no function_ref refs, no valueRef edges.
|
||||
expect(viaKernel!.unresolvedReferences.some((r) => r.referenceKind === 'function_ref')).toBe(
|
||||
false
|
||||
);
|
||||
expect(viaKernel!.edges.some((e) => e.metadata?.valueRef === true)).toBe(false);
|
||||
});
|
||||
|
||||
it('empty object patterns defer (the dominant dart-3 error class)', () => {
|
||||
const broken = 'int f(Object x) => switch (x) { Init() => 1, _ => 0 };\n';
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('lib/pat.dart', broken, 'dart')).toBeNull();
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource('lib/pat.dart', broken, 'dart');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(viaWasm.nodes.some((n) => n.kind === 'file')).toBe(true);
|
||||
});
|
||||
|
||||
it('unnamed `library;` defers', () => {
|
||||
const broken = '/// Doc.\nlibrary;\n\nvoid f() {}\n';
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('lib/lib.dart', broken, 'dart')).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -36,7 +36,7 @@ const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
// Every kernel-capable language. `jsx` shares the javascript grammar on BOTH
|
||||
// paths (langs.rs mirrors WASM_GRAMMAR_FILES), so the distinct grammars are:
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau', 'scala'];
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau', 'scala', 'dart'];
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel↔wasm grammar parity', () => {
|
||||
beforeAll(async () => {
|
||||
|
||||
@@ -60,4 +60,28 @@ fn main() {
|
||||
scala.flag_if_supported("-utf-8"); // msvc
|
||||
scala.compile("tree-sitter-scala");
|
||||
println!("cargo:rerun-if-changed=grammars/scala");
|
||||
|
||||
// Dart grammar — vendored C (fourth vendored-grammar-C language): the
|
||||
// production wasm is the tree-sitter-wasms 0.1.13 artifact, whose dart
|
||||
// dependency is an UNPINNED github:UserNobody14/tree-sitter-dart — the
|
||||
// vendored wasm byte-copy (src/extraction/wasm/) plus these same-commit
|
||||
// sources kill that hazard. crates.io tree-sitter-dart is the nielsenko
|
||||
// FORK (different lineage) — rejected. Sources are
|
||||
// UserNobody14/tree-sitter-dart master@d4d8f3e337d8 (dart checklist
|
||||
// §Grammar prep):
|
||||
// parser.c 5a42b47abb4d494f125dbdee9138979248041689b1aa36355550fa3e28dcb8b8
|
||||
// scanner.c 07a7b7818b175e9460523e705dd88d20f7b5141bac95c593d4426e6d52284996
|
||||
// scanner.c is load-bearing: string-template char classes and /** */ doc
|
||||
// comments are external tokens.
|
||||
let mut dart = cc::Build::new();
|
||||
dart.include("grammars/dart");
|
||||
dart.file("grammars/dart/parser.c");
|
||||
dart.file("grammars/dart/scanner.c");
|
||||
dart.flag_if_supported("-Wno-unused-parameter");
|
||||
dart.flag_if_supported("-Wno-unused-but-set-variable");
|
||||
dart.flag_if_supported("-Wno-trigraphs");
|
||||
dart.flag_if_supported("-Wno-unused-function");
|
||||
dart.flag_if_supported("-utf-8"); // msvc
|
||||
dart.compile("tree-sitter-dart");
|
||||
println!("cargo:rerun-if-changed=grammars/dart");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
Copyright (c) 2020-2023 UserNobody14 and others
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of this software and associated documentation files (the
|
||||
"Software"), to deal in the Software without restriction, including
|
||||
without limitation the rights to use, copy, modify, merge, publish,
|
||||
distribute, sublicense, and/or sell copies of the Software, and to
|
||||
permit persons to whom the Software is furnished to do so, subject to
|
||||
the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be
|
||||
included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
||||
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
||||
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
||||
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,130 @@
|
||||
#include "tree_sitter/parser.h"
|
||||
#include <wctype.h>
|
||||
|
||||
enum TokenType {
|
||||
TEMPLATE_CHARS_SINGLE,
|
||||
TEMPLATE_CHARS_DOUBLE,
|
||||
TEMPLATE_CHARS_SINGLE_SINGLE,
|
||||
TEMPLATE_CHARS_DOUBLE_SINGLE,
|
||||
TEMPLATE_CHARS_RAW_SLASH,
|
||||
BLOCK_COMMENT,
|
||||
DOCUMENTATION_BLOCK_COMMENT,
|
||||
};
|
||||
|
||||
void *tree_sitter_dart_external_scanner_create() { return NULL; }
|
||||
void tree_sitter_dart_external_scanner_destroy(void *p) {}
|
||||
void tree_sitter_dart_external_scanner_reset(void *p) {}
|
||||
unsigned tree_sitter_dart_external_scanner_serialize(void *p, char *buffer) { return 0; }
|
||||
void tree_sitter_dart_external_scanner_deserialize(void *p, const char *b, unsigned n) {}
|
||||
|
||||
static void advance(TSLexer *lexer) { lexer->advance(lexer, false); }
|
||||
static void skip(TSLexer *lexer) { lexer->advance(lexer, true); }
|
||||
|
||||
static bool scan_multiline_comments(TSLexer *lexer) {
|
||||
|
||||
bool documentation_comment = false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '*') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '*') documentation_comment = true;
|
||||
|
||||
bool after_star = false;
|
||||
unsigned nesting_depth = 1;
|
||||
for (;;) {
|
||||
switch (lexer->lookahead) {
|
||||
case '\0':
|
||||
return false;
|
||||
case '*':
|
||||
advance(lexer);
|
||||
after_star = true;
|
||||
break;
|
||||
case '/':
|
||||
if (after_star) {
|
||||
advance(lexer);
|
||||
after_star = false;
|
||||
nesting_depth--;
|
||||
if (nesting_depth == 0) {
|
||||
if (!documentation_comment) {
|
||||
lexer->result_symbol = BLOCK_COMMENT;
|
||||
} else {
|
||||
lexer->result_symbol = DOCUMENTATION_BLOCK_COMMENT;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
} else {
|
||||
advance(lexer);
|
||||
after_star = false;
|
||||
if (lexer->lookahead == '*') {
|
||||
nesting_depth++;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
break;
|
||||
default:
|
||||
advance(lexer);
|
||||
after_star = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
static bool scan_templates(TSLexer *lexer, const bool *valid_symbols) {
|
||||
if(valid_symbols[TEMPLATE_CHARS_DOUBLE]) {
|
||||
lexer->result_symbol = TEMPLATE_CHARS_DOUBLE;
|
||||
} else if (valid_symbols[TEMPLATE_CHARS_SINGLE]) {
|
||||
lexer->result_symbol = TEMPLATE_CHARS_SINGLE;
|
||||
} else if (valid_symbols[TEMPLATE_CHARS_SINGLE_SINGLE]) {
|
||||
lexer->result_symbol = TEMPLATE_CHARS_SINGLE_SINGLE;
|
||||
} else {
|
||||
lexer->result_symbol = TEMPLATE_CHARS_DOUBLE_SINGLE;
|
||||
}
|
||||
for (bool has_content = false;; has_content = true) {
|
||||
lexer->mark_end(lexer);
|
||||
switch (lexer->lookahead) {
|
||||
case '\'':
|
||||
case '"':
|
||||
return has_content;
|
||||
case '\n':
|
||||
if (valid_symbols[TEMPLATE_CHARS_DOUBLE_SINGLE] || valid_symbols[TEMPLATE_CHARS_SINGLE_SINGLE]) return false;
|
||||
advance(lexer);
|
||||
break;
|
||||
case '\0':
|
||||
return false;
|
||||
case '$':
|
||||
return has_content;
|
||||
case '\\':
|
||||
if (valid_symbols[TEMPLATE_CHARS_RAW_SLASH]) {
|
||||
lexer->result_symbol = TEMPLATE_CHARS_RAW_SLASH;
|
||||
advance(lexer);
|
||||
} else {
|
||||
return has_content;
|
||||
}
|
||||
break;
|
||||
default:
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
|
||||
bool tree_sitter_dart_external_scanner_scan(void *payload, TSLexer *lexer,
|
||||
const bool *valid_symbols) {
|
||||
if (
|
||||
valid_symbols[TEMPLATE_CHARS_DOUBLE] ||
|
||||
valid_symbols[TEMPLATE_CHARS_SINGLE] ||
|
||||
valid_symbols[TEMPLATE_CHARS_DOUBLE_SINGLE] ||
|
||||
valid_symbols[TEMPLATE_CHARS_SINGLE_SINGLE]
|
||||
) {
|
||||
return scan_templates(lexer, valid_symbols);
|
||||
}
|
||||
while (iswspace(lexer->lookahead)) lexer->advance(lexer, true);
|
||||
|
||||
if (lexer->lookahead == '/') {
|
||||
return scan_multiline_comments(lexer);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
#ifndef TREE_SITTER_ALLOC_H_
|
||||
#define TREE_SITTER_ALLOC_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
// Allow clients to override allocation functions
|
||||
#ifdef TREE_SITTER_REUSE_ALLOCATOR
|
||||
|
||||
extern void *(*ts_current_malloc)(size_t size);
|
||||
extern void *(*ts_current_calloc)(size_t count, size_t size);
|
||||
extern void *(*ts_current_realloc)(void *ptr, size_t size);
|
||||
extern void (*ts_current_free)(void *ptr);
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc ts_current_malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc ts_current_calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc ts_current_realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free ts_current_free
|
||||
#endif
|
||||
|
||||
#else
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free free
|
||||
#endif
|
||||
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ALLOC_H_
|
||||
@@ -0,0 +1,291 @@
|
||||
#ifndef TREE_SITTER_ARRAY_H_
|
||||
#define TREE_SITTER_ARRAY_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include "./alloc.h"
|
||||
|
||||
#include <assert.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(push)
|
||||
#pragma warning(disable : 4101)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-variable"
|
||||
#endif
|
||||
|
||||
#define Array(T) \
|
||||
struct { \
|
||||
T *contents; \
|
||||
uint32_t size; \
|
||||
uint32_t capacity; \
|
||||
}
|
||||
|
||||
/// Initialize an array.
|
||||
#define array_init(self) \
|
||||
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
|
||||
|
||||
/// Create an empty array.
|
||||
#define array_new() \
|
||||
{ NULL, 0, 0 }
|
||||
|
||||
/// Get a pointer to the element at a given `index` in the array.
|
||||
#define array_get(self, _index) \
|
||||
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
|
||||
|
||||
/// Get a pointer to the first element in the array.
|
||||
#define array_front(self) array_get(self, 0)
|
||||
|
||||
/// Get a pointer to the last element in the array.
|
||||
#define array_back(self) array_get(self, (self)->size - 1)
|
||||
|
||||
/// Clear the array, setting its size to zero. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_delete(self) _array__delete((Array *)(self))
|
||||
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
|
||||
(self)->contents[(self)->size++] = (element))
|
||||
|
||||
/// Increase the array's size by `count` elements.
|
||||
/// New elements are zero-initialized.
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
_array__grow((Array *)(self), count, array_elem_size(self)); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
|
||||
/// Append all elements from one array to the end of another.
|
||||
#define array_push_all(self, other) \
|
||||
array_extend((self), (other)->size, (other)->contents)
|
||||
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), (self)->size, \
|
||||
0, count, contents \
|
||||
)
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), _index, \
|
||||
old_count, new_count, new_contents \
|
||||
)
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
_array__erase((Array *)(self), array_elem_size(self), _index)
|
||||
|
||||
/// Pop the last element off the array, returning the element by value.
|
||||
#define array_pop(self) ((self)->contents[--(self)->size])
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
_array__swap((Array *)(self), (Array *)(other))
|
||||
|
||||
/// Get the size of the array contents
|
||||
#define array_elem_size(self) (sizeof *(self)->contents)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
///
|
||||
/// If an existing element is found to be equal to `needle`, then the `index`
|
||||
/// out-parameter is set to the existing value's index, and the `exists`
|
||||
/// out-parameter is set to true. Otherwise, `index` is set to an index where
|
||||
/// `needle` should be inserted in order to preserve the sorting, and `exists`
|
||||
/// is set to false.
|
||||
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using integer comparisons
|
||||
/// of a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_with`.
|
||||
#define array_search_sorted_by(self, field, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
#define array_insert_sorted_with(self, compare, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using integer comparisons of
|
||||
/// a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_by`.
|
||||
#define array_insert_sorted_by(self, field, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
// Private
|
||||
|
||||
typedef Array(void) Array;
|
||||
|
||||
/// This is not what you're looking for, see `array_delete`.
|
||||
static inline void _array__delete(Array *self) {
|
||||
if (self->contents) {
|
||||
ts_free(self->contents);
|
||||
self->contents = NULL;
|
||||
self->size = 0;
|
||||
self->capacity = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_erase`.
|
||||
static inline void _array__erase(Array *self, size_t element_size,
|
||||
uint32_t index) {
|
||||
assert(index < self->size);
|
||||
char *contents = (char *)self->contents;
|
||||
memmove(contents + index * element_size, contents + (index + 1) * element_size,
|
||||
(self->size - index - 1) * element_size);
|
||||
self->size--;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_reserve`.
|
||||
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
|
||||
if (new_capacity > self->capacity) {
|
||||
if (self->contents) {
|
||||
self->contents = ts_realloc(self->contents, new_capacity * element_size);
|
||||
} else {
|
||||
self->contents = ts_malloc(new_capacity * element_size);
|
||||
}
|
||||
self->capacity = new_capacity;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_assign`.
|
||||
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
|
||||
_array__reserve(self, element_size, other->size);
|
||||
self->size = other->size;
|
||||
memcpy(self->contents, other->contents, self->size * element_size);
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_swap`.
|
||||
static inline void _array__swap(Array *self, Array *other) {
|
||||
Array swap = *other;
|
||||
*other = *self;
|
||||
*self = swap;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
|
||||
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
|
||||
uint32_t new_size = self->size + count;
|
||||
if (new_size > self->capacity) {
|
||||
uint32_t new_capacity = self->capacity * 2;
|
||||
if (new_capacity < 8) new_capacity = 8;
|
||||
if (new_capacity < new_size) new_capacity = new_size;
|
||||
_array__reserve(self, element_size, new_capacity);
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_splice`.
|
||||
static inline void _array__splice(Array *self, size_t element_size,
|
||||
uint32_t index, uint32_t old_count,
|
||||
uint32_t new_count, const void *elements) {
|
||||
uint32_t new_size = self->size + new_count - old_count;
|
||||
uint32_t old_end = index + old_count;
|
||||
uint32_t new_end = index + new_count;
|
||||
assert(old_end <= self->size);
|
||||
|
||||
_array__reserve(self, element_size, new_size);
|
||||
|
||||
char *contents = (char *)self->contents;
|
||||
if (self->size > old_end) {
|
||||
memmove(
|
||||
contents + new_end * element_size,
|
||||
contents + old_end * element_size,
|
||||
(self->size - old_end) * element_size
|
||||
);
|
||||
}
|
||||
if (new_count > 0) {
|
||||
if (elements) {
|
||||
memcpy(
|
||||
(contents + index * element_size),
|
||||
elements,
|
||||
new_count * element_size
|
||||
);
|
||||
} else {
|
||||
memset(
|
||||
(contents + index * element_size),
|
||||
0,
|
||||
new_count * element_size
|
||||
);
|
||||
}
|
||||
}
|
||||
self->size += new_count - old_count;
|
||||
}
|
||||
|
||||
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
|
||||
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
|
||||
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
|
||||
do { \
|
||||
*(_index) = start; \
|
||||
*(_exists) = false; \
|
||||
uint32_t size = (self)->size - *(_index); \
|
||||
if (size == 0) break; \
|
||||
int comparison; \
|
||||
while (size > 1) { \
|
||||
uint32_t half_size = size / 2; \
|
||||
uint32_t mid_index = *(_index) + half_size; \
|
||||
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
|
||||
if (comparison <= 0) *(_index) = mid_index; \
|
||||
size -= half_size; \
|
||||
} \
|
||||
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
|
||||
if (comparison == 0) *(_exists) = true; \
|
||||
else if (comparison < 0) *(_index) += 1; \
|
||||
} while (0)
|
||||
|
||||
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
|
||||
/// parameter by reference in order to work with the generic sorting function above.
|
||||
#define _compare_int(a, b) ((int)*(a) - (int)(b))
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(pop)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ARRAY_H_
|
||||
@@ -0,0 +1,286 @@
|
||||
#ifndef TREE_SITTER_PARSER_H_
|
||||
#define TREE_SITTER_PARSER_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
#define ts_builtin_sym_error ((TSSymbol)-1)
|
||||
#define ts_builtin_sym_end 0
|
||||
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
|
||||
|
||||
#ifndef TREE_SITTER_API_H_
|
||||
typedef uint16_t TSStateId;
|
||||
typedef uint16_t TSSymbol;
|
||||
typedef uint16_t TSFieldId;
|
||||
typedef struct TSLanguage TSLanguage;
|
||||
typedef struct TSLanguageMetadata {
|
||||
uint8_t major_version;
|
||||
uint8_t minor_version;
|
||||
uint8_t patch_version;
|
||||
} TSLanguageMetadata;
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
TSFieldId field_id;
|
||||
uint8_t child_index;
|
||||
bool inherited;
|
||||
} TSFieldMapEntry;
|
||||
|
||||
// Used to index the field and supertype maps.
|
||||
typedef struct {
|
||||
uint16_t index;
|
||||
uint16_t length;
|
||||
} TSMapSlice;
|
||||
|
||||
typedef struct {
|
||||
bool visible;
|
||||
bool named;
|
||||
bool supertype;
|
||||
} TSSymbolMetadata;
|
||||
|
||||
typedef struct TSLexer TSLexer;
|
||||
|
||||
struct TSLexer {
|
||||
int32_t lookahead;
|
||||
TSSymbol result_symbol;
|
||||
void (*advance)(TSLexer *, bool);
|
||||
void (*mark_end)(TSLexer *);
|
||||
uint32_t (*get_column)(TSLexer *);
|
||||
bool (*is_at_included_range_start)(const TSLexer *);
|
||||
bool (*eof)(const TSLexer *);
|
||||
void (*log)(const TSLexer *, const char *, ...);
|
||||
};
|
||||
|
||||
typedef enum {
|
||||
TSParseActionTypeShift,
|
||||
TSParseActionTypeReduce,
|
||||
TSParseActionTypeAccept,
|
||||
TSParseActionTypeRecover,
|
||||
} TSParseActionType;
|
||||
|
||||
typedef union {
|
||||
struct {
|
||||
uint8_t type;
|
||||
TSStateId state;
|
||||
bool extra;
|
||||
bool repetition;
|
||||
} shift;
|
||||
struct {
|
||||
uint8_t type;
|
||||
uint8_t child_count;
|
||||
TSSymbol symbol;
|
||||
int16_t dynamic_precedence;
|
||||
uint16_t production_id;
|
||||
} reduce;
|
||||
uint8_t type;
|
||||
} TSParseAction;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
} TSLexMode;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
uint16_t reserved_word_set_id;
|
||||
} TSLexerMode;
|
||||
|
||||
typedef union {
|
||||
TSParseAction action;
|
||||
struct {
|
||||
uint8_t count;
|
||||
bool reusable;
|
||||
} entry;
|
||||
} TSParseActionEntry;
|
||||
|
||||
typedef struct {
|
||||
int32_t start;
|
||||
int32_t end;
|
||||
} TSCharacterRange;
|
||||
|
||||
struct TSLanguage {
|
||||
uint32_t abi_version;
|
||||
uint32_t symbol_count;
|
||||
uint32_t alias_count;
|
||||
uint32_t token_count;
|
||||
uint32_t external_token_count;
|
||||
uint32_t state_count;
|
||||
uint32_t large_state_count;
|
||||
uint32_t production_id_count;
|
||||
uint32_t field_count;
|
||||
uint16_t max_alias_sequence_length;
|
||||
const uint16_t *parse_table;
|
||||
const uint16_t *small_parse_table;
|
||||
const uint32_t *small_parse_table_map;
|
||||
const TSParseActionEntry *parse_actions;
|
||||
const char * const *symbol_names;
|
||||
const char * const *field_names;
|
||||
const TSMapSlice *field_map_slices;
|
||||
const TSFieldMapEntry *field_map_entries;
|
||||
const TSSymbolMetadata *symbol_metadata;
|
||||
const TSSymbol *public_symbol_map;
|
||||
const uint16_t *alias_map;
|
||||
const TSSymbol *alias_sequences;
|
||||
const TSLexerMode *lex_modes;
|
||||
bool (*lex_fn)(TSLexer *, TSStateId);
|
||||
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
|
||||
TSSymbol keyword_capture_token;
|
||||
struct {
|
||||
const bool *states;
|
||||
const TSSymbol *symbol_map;
|
||||
void *(*create)(void);
|
||||
void (*destroy)(void *);
|
||||
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
|
||||
unsigned (*serialize)(void *, char *);
|
||||
void (*deserialize)(void *, const char *, unsigned);
|
||||
} external_scanner;
|
||||
const TSStateId *primary_state_ids;
|
||||
const char *name;
|
||||
const TSSymbol *reserved_words;
|
||||
uint16_t max_reserved_word_set_size;
|
||||
uint32_t supertype_count;
|
||||
const TSSymbol *supertype_symbols;
|
||||
const TSMapSlice *supertype_map_slices;
|
||||
const TSSymbol *supertype_map_entries;
|
||||
TSLanguageMetadata metadata;
|
||||
};
|
||||
|
||||
static inline bool set_contains(const TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
|
||||
uint32_t index = 0;
|
||||
uint32_t size = len - index;
|
||||
while (size > 1) {
|
||||
uint32_t half_size = size / 2;
|
||||
uint32_t mid_index = index + half_size;
|
||||
const TSCharacterRange *range = &ranges[mid_index];
|
||||
if (lookahead >= range->start && lookahead <= range->end) {
|
||||
return true;
|
||||
} else if (lookahead > range->end) {
|
||||
index = mid_index;
|
||||
}
|
||||
size -= half_size;
|
||||
}
|
||||
const TSCharacterRange *range = &ranges[index];
|
||||
return (lookahead >= range->start && lookahead <= range->end);
|
||||
}
|
||||
|
||||
/*
|
||||
* Lexer Macros
|
||||
*/
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#define UNUSED __pragma(warning(suppress : 4101))
|
||||
#else
|
||||
#define UNUSED __attribute__((unused))
|
||||
#endif
|
||||
|
||||
#define START_LEXER() \
|
||||
bool result = false; \
|
||||
bool skip = false; \
|
||||
UNUSED \
|
||||
bool eof = false; \
|
||||
int32_t lookahead; \
|
||||
goto start; \
|
||||
next_state: \
|
||||
lexer->advance(lexer, skip); \
|
||||
start: \
|
||||
skip = false; \
|
||||
lookahead = lexer->lookahead;
|
||||
|
||||
#define ADVANCE(state_value) \
|
||||
{ \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ADVANCE_MAP(...) \
|
||||
{ \
|
||||
static const uint16_t map[] = { __VA_ARGS__ }; \
|
||||
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
|
||||
if (map[i] == lookahead) { \
|
||||
state = map[i + 1]; \
|
||||
goto next_state; \
|
||||
} \
|
||||
} \
|
||||
}
|
||||
|
||||
#define SKIP(state_value) \
|
||||
{ \
|
||||
skip = true; \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ACCEPT_TOKEN(symbol_value) \
|
||||
result = true; \
|
||||
lexer->result_symbol = symbol_value; \
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
#define END_STATE() return result;
|
||||
|
||||
/*
|
||||
* Parse Table Macros
|
||||
*/
|
||||
|
||||
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
|
||||
|
||||
#define STATE(id) id
|
||||
|
||||
#define ACTIONS(id) id
|
||||
|
||||
#define SHIFT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value) \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_REPEAT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value), \
|
||||
.repetition = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_EXTRA() \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.extra = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define REDUCE(symbol_name, children, precedence, prod_id) \
|
||||
{{ \
|
||||
.reduce = { \
|
||||
.type = TSParseActionTypeReduce, \
|
||||
.symbol = symbol_name, \
|
||||
.child_count = children, \
|
||||
.dynamic_precedence = precedence, \
|
||||
.production_id = prod_id \
|
||||
}, \
|
||||
}}
|
||||
|
||||
#define RECOVER() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeRecover \
|
||||
}}
|
||||
|
||||
#define ACCEPT_INPUT() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeAccept \
|
||||
}}
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_PARSER_H_
|
||||
File diff suppressed because it is too large
Load Diff
@@ -28,12 +28,17 @@ extern "C" {
|
||||
extern "C" {
|
||||
fn tree_sitter_scala() -> *const ();
|
||||
}
|
||||
// Vendored dart grammar (build.rs-compiled C — UserNobody14 d4d8f3e; the
|
||||
// crates.io crate is a different-lineage fork; see grammars/dart).
|
||||
extern "C" {
|
||||
fn tree_sitter_dart() -> *const ();
|
||||
}
|
||||
|
||||
/// Languages this kernel binary can extract (reported by contractInfo;
|
||||
/// TS-side routing policy decides what actually routes).
|
||||
pub const LANGUAGES: [&str; 19] = [
|
||||
pub const LANGUAGES: [&str; 20] = [
|
||||
"typescript", "tsx", "javascript", "jsx", "java", "python", "go", "c", "cpp", "rust",
|
||||
"csharp", "ruby", "php", "swift", "kotlin", "r", "lua", "luau", "scala",
|
||||
"csharp", "ruby", "php", "swift", "kotlin", "r", "lua", "luau", "scala", "dart",
|
||||
];
|
||||
|
||||
pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
@@ -83,6 +88,11 @@ pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
"scala" => {
|
||||
Some(unsafe { tree_sitter_language::LanguageFn::from_raw(tree_sitter_scala) }.into())
|
||||
}
|
||||
// R7b batch 4: UserNobody14 d4d8f3e vendored C compiled in build.rs
|
||||
// (same commit as the byte-copied tree-sitter-wasms 0.1.13 artifact).
|
||||
"dart" => {
|
||||
Some(unsafe { tree_sitter_language::LanguageFn::from_raw(tree_sitter_dart) }.into())
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,6 +20,7 @@ mod buffers;
|
||||
mod ccpp;
|
||||
mod cfnptr;
|
||||
mod csharp;
|
||||
mod dart;
|
||||
mod docstring;
|
||||
mod ids;
|
||||
mod go;
|
||||
@@ -229,6 +230,7 @@ pub fn extract_file(file_path: String, content: String, language: String) -> Res
|
||||
"r" => rlang::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"lua" | "luau" => lua::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
|
||||
"scala" => scala::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"dart" => dart::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
_ => tsjs::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
|
||||
};
|
||||
Ok(ExtractBuffers {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -611,7 +611,7 @@ parity before porting the language.
|
||||
| r | `languages/r.ts` | T1 | crates.io | **DONE (R7b batch 4 #1, 2026-07-20)** — `rlang.rs` walker; NO grammar change (crate pin `=1.2.0`: the crates.io tarball ships parser.c AND scanner.c sha-identical to the r-lib v1.2.0 tag the vendored wasm was built from — first true no-op grammar prep). The lightest-shared-surface, heaviest-hook port: r.ts works entirely through visitNode (every type list empty but `callTypes`), four shared machineries dead by language gates, walker = file node + hook transcription + generic extractCall. Parity 0-diff on AnomalyDetection/dplyr/ggplot2/shiny (838 files, deferrals 0/0/0/1 — the 1 = a moustache-template pseudo-R file, both-arm) + dump byte-identical ×3 (dplyr/ggplot2/shiny). kernel-parity.mjs gained lowercased-extension matching (`.R` is the dominant casing). Quirk list: docs/design/r-kernel-port-checklist.md. | ☑ |
|
||||
| lua, luau | `languages/lua.ts` + `luau.ts` (36-line extension) | T1 | lua: **vendored C** (v0.4.1 not on crates.io); luau: crates.io `=1.2.0` (tag≡crate sha-verified) | **DONE (R7b batch 4 #2, 2026-07-20)** — ONE walker (`lua.rs`, ccpp-style dialect flag); lua = the second vendored-grammar-C language (v0.4.1 tag artifacts, shas in the checklist); luau = plain crate pin. NO wasm change for either — grammar-parity rows replace the bump gate. Ports the require/visitNode-hook asymmetries (top-level imports vs body `calls "require"`), receiver-QN methods, the top-level initializer-visibility inversion, raw-text callee world (colon/bracket/glue chains, paren-conversion), LUA_SPEC fn-ref capture, LuaDoc `- `-keeping docstrings, and the lua↔luau isExported wire divergence. Parity 0-diff kong/lazy.nvim/lua-resty-core/lune/Fusion (1,734 clean files; deferrals 1/0/0/3/8 — every one matching the survey's both-arm predictions) + dump byte-identical ×4 (kong 157,650 dump lines). Quirk list: docs/design/lua-luau-kernel-port-checklist.md. | ☑ |
|
||||
| scala | `languages/scala.ts` | T1 | **vendored C** (master@0aca5d0a6f — not a release; crate 0.26.0 is 30 states behind) | **DONE (R7b batch 4 #3, 2026-07-20)** — `scala.rs` walker; the third vendored-grammar-C language (35MB parser.c — the biggest grammar in the tree). NO wasm change (production has parsed with this exact revision since #91) — the grammar-parity row is the whole alignment proof. Ports the leak-through asymmetries (extension first-def call leak + braced-form invisibility via the `{`-token body field, anon `new T {…}` template_body member leaks, bodied-vs-bodiless class_parameters), first-segment import names, the val/var hook's enclosing-NODE-TYPE kinds, defs-as-methods with top-level function fallback, nested-def invisibility, curried/type-params-first signatures, the #750 capitalized re-encode, static-member WRITES, scaladoc retention, full value-refs (last-wins targets) + SCALA_SPEC fn-refs (varinit + postfix eta). Parity 0-diff os-lib/cats/scala3-compiler-src/scala3-library-src (1,935 clean files; deferrals 0/15/57/116 — every count matching the survey exactly; scala-3 PHANTOM hasError deferred on the flag) + dump byte-identical ×3 (scala3 whole-repo 950,889 dump lines). Quirk list: docs/design/scala-kernel-port-checklist.md. | ☑ |
|
||||
| dart | `languages/dart.ts` | T1 | **vendored C** (UserNobody14 d4d8f3e — reached via tree-sitter-wasms' unpinned github dep) | Long-tail T1, survey COMPLETE 2026-07-20 (docs/design/dart-kernel-port-checklist.md). Port = vendored-C + byte-copy the tree-sitter-wasms wasm artifact into src/extraction/wasm/ (kills the unpinned-dep hazard). The sibling-body DOUBLE-WALK (duplicate ids, ×3 refs) is the port hazard; sweeps --max-deferral 0.3. | ☐ |
|
||||
| dart | `languages/dart.ts` | T1 | **vendored C** (UserNobody14 d4d8f3e; wasm = the byte-copied tree-sitter-wasms 0.1.13 artifact, same commit) | **DONE (R7b batch 4 #4, 2026-07-20 — the FINAL R7b language)** — `dart.rs` walker; the fourth vendored-grammar-C language. The wasm byte-copy into src/extraction/wasm/ + VENDORED_WASM_LANGS kills tree-sitter-wasms' UNPINNED github-dep hazard (crates.io dart is a different-lineage fork — rejected). Ports THE SIBLING-BODY DOUBLE-WALK bug-for-bug (duplicate local-fn nodes sharing an id under different parents, duplicated calls/instantiates, file/class fn-ref twins — pinned by a dedicated fixture + the bloc kind-census spot-check), the extractBareCall selector matrix (first callTypes=[] language; cascades invisible, `?.`≡`.`, the `ConfigT.load()` calls+references double emission, capitalized-chain re-encode), the constructor naming/skip hooks (unnamed ctor skipped; named ctors renamed with class-as-returnType), operator methods as `<anonymous>`, static_final_declaration constants via the hook (instance fields mint nothing), prefixed-return-type prefix bug, enum-with silence, anonymous extensions named after the ON type, deferred-import invisibility, named-arg fn-ref non-capture, async*/sync*≠async, value-refs with the LIVE sibling-body pull. Parity 0-diff shelf/bloc/flutter (5,815 clean files; deferrals 10/21/1341 ≈ the survey's 10/21/~1340 — both-arm empty-object-pattern + `library;` reality, --max-deferral 0.3) + dump byte-identical ×3 (flutter 6,472 dart files) + bloc census identical per kind. Quirk list: docs/design/dart-kernel-port-checklist.md. | ☑ |
|
||||
| kotlin | `languages/kotlin.ts` | T1½ | **vendored C** (crate unusable) | **DONE (R7b #6, 2026-07-20)** — `kotlin.rs` walker; the arc's FIRST vendored-grammar-C language: fwcd 0.3.8's sha-matched parser.c/scanner.c compile inside codegraph-kernel via build.rs + cc (the crates.io crate pins tree-sitter <0.23; tree-sitter-kotlin-ng is a different grammar). Behavior-neutral wasm re-vendor (dumps byte-identical old-vs-new ×3). Two walker firsts: extension-fn receiver QNs + owner-contains, and extractModifiers→decorators (KMP expect/actual — 412 synthesized edges identical both arms on kotlinx.coroutines). Parity 0-diff okio/okhttp/kotlinx.coroutines (1,861 clean files; deferral 4.7–8.5% both-arm incl. PHANTOM hasError files). Quirk list: docs/design/kotlin-kernel-port-checklist.md. | ☑ |
|
||||
| swift | shared + dedicated branch | T1½ | crates.io | **DONE (R7b #5, 2026-07-20)** — `swift.rs` walker incl. the #1020 dedicated property branch (Alamofire's 348 property nodes reproduced exactly on the kernel arm); grammar bumped to crate 0.7.3 (wasm built from the CRATE TARBALL's src — the tag ships an older ABI-14 generation; delta = error-set membership + 2 gate-found categories, all classified). Parity 0-diff Alamofire/vapor/swift-nio (720 clean files; deferral 9–27% both-arm structural — sweeps use --max-deferral 0.3). One walker fix found by the sweep: the shared `assignment` shadow-prune case is swift-live (declared-then-assigned `let X: T`). Quirk list: docs/design/swift-kernel-port-checklist.md. | ☑ |
|
||||
| c, cpp | `languages/c-cpp.ts` | **T2** | crates.io | **DONE (R7a, 2026-07-17)** — `ccpp/` walker; ALL pre-passes stayed TS-side via the route-point preParse hoist (+6 new blanks added during gating — see the checklist doc); content-based `.h` C-vs-C++ detection stays upstream at detectLanguage. Parity 0-diff + dump byte-identical on redis/git/fmt/protobuf/ALS. | ☑ |
|
||||
|
||||
@@ -48,7 +48,7 @@ if (paths.length === 0) {
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau', 'scala']);
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau', 'scala', 'dart']);
|
||||
const EXTS = new Map([
|
||||
['.ts', 'typescript'], ['.mts', 'typescript'], ['.cts', 'typescript'],
|
||||
['.tsx', 'tsx'], ['.js', 'javascript'], ['.mjs', 'javascript'],
|
||||
@@ -67,6 +67,7 @@ const EXTS = new Map([
|
||||
['.r', 'r'], // R7b batch 4 — real-world casing is `.R`; extname lowercased below
|
||||
['.lua', 'lua'], ['.luau', 'luau'], // R7b batch 4
|
||||
['.scala', 'scala'], ['.sc', 'scala'], // R7b batch 4
|
||||
['.dart', 'dart'], // R7b batch 4
|
||||
]);
|
||||
|
||||
/** Collect candidate files. */
|
||||
|
||||
@@ -330,6 +330,14 @@ const VENDORED_WASM_LANGS: ReadonlySet<GrammarLanguage> = new Set([
|
||||
// tree-sitter-kotlin-ng is a different grammar — the kernel compiles the
|
||||
// same vendored C sources instead (codegraph-kernel/grammars/kotlin).
|
||||
'kotlin',
|
||||
// R7b batch 4 (Dart kernel port prep): the byte-copied tree-sitter-wasms
|
||||
// 0.1.13 artifact (sha256 7f5364e4…, built from UserNobody14/
|
||||
// tree-sitter-dart master@d4d8f3e337d8). tree-sitter-wasms' dart dep is an
|
||||
// UNPINNED github ref, so a routine tree-sitter-wasms update would have
|
||||
// silently changed dart's grammar — vendoring kills that hazard. The
|
||||
// kernel compiles the same-commit vendored C (codegraph-kernel/grammars/
|
||||
// dart); crates.io tree-sitter-dart is a different-lineage fork (rejected).
|
||||
'dart',
|
||||
]);
|
||||
|
||||
/** Absolute path of a language's grammar WASM (vendored or tree-sitter-wasms). */
|
||||
|
||||
@@ -103,6 +103,13 @@ const DEFAULT_ROUTED: ReadonlySet<Language> = new Set<Language>([
|
||||
// scala3-style repos use --max-deferral 0.3 (swift precedent); a
|
||||
// deferral JUMP on cats/os-lib-style code is the bug signal.
|
||||
'scala',
|
||||
// R7b batch 4 (2026-07-20): parity swept 0-diff on shelf/bloc/flutter +
|
||||
// full-init dump-diffs byte-identical. Error incidence 3.4-20.7% both-arm
|
||||
// grammar reality (empty object patterns — the sealed-class idiom — and
|
||||
// unnamed `library;` dominate; flutter sits at ~21%) — sweeps use
|
||||
// --max-deferral 0.3; the RATE, not the count, is the signal (flutter
|
||||
// HEAD drifts).
|
||||
'dart',
|
||||
]);
|
||||
|
||||
/**
|
||||
|
||||
Executable
BIN
Binary file not shown.
Reference in New Issue
Block a user