feat: migrate to alef polyglot binding generator by Goldziher · Pull Request #105 · kreuzberg-dev/tree-sitter-language-pack

Goldziher · 2026-04-21T07:17:17Z

Summary

Migrate all polyglot bindings from hand-written crates to alef-generated code
Replace legacy tooling (e2e-generator, generate_readme.py, sync_versions.py) with alef CLI
Add alef-generated API reference docs, README generation, and version sync
Add 16 new e2e fixtures covering all previously untested API functions
Wire alef into CI, pre-commit hooks, and Taskfile

What changed

Bindings (alef-generated):

Python (crates/ts-pack-core-py/) — PyO3 bindings
Node (crates/ts-pack-core-node/) — NAPI-RS bindings
PHP (crates/ts-pack-core-php/) — ext-php-rs bindings
FFI (crates/ts-pack-core-ffi/) — C FFI layer
WASM (crates/ts-pack-core-wasm/) — wasm-bindgen bindings
Go, Java, C#, Elixir, Ruby packages regenerated

Core crate changes:

Add serde derives to QueryMatch (behind feature flag)
Add ValidationResult, PatternValidation, MatchResult, PatternResult to public API
DownloadManager uses interior mutability (Mutex<Option<T>>) for &self methods
Add Default/Hash/PartialEq/Eq derives to 30+ public types

Tooling migration:

Delete tools/e2e-generator/ (replaced by alef e2e generate)
Delete scripts/generate_readme.py, scripts/sync_versions.py, scripts/convert_fixtures.py
Delete scripts/readme_templates/
Keep tools/snippet-runner/ for doc snippet validation

Documentation:

Replace hand-written docs/api/ (12 files) with alef-generated docs/reference/ (14 files)
Rust API docs now included
Tuple types render correctly (e.g., tuple[str, list[str]] for Python)
Update zensical.toml nav to reference/ paths, align with kreuzberg

E2E testing:

393 fixtures (16 new), all pass schema validation
Rust e2e: 162 tests pass, 0 failures
New fixtures for: highlights, injections, locals, run_query, tree_to_sexp, tree_has_error_nodes, extract_patterns, validate_extraction, language_count, extension_ambiguity

CI:

New alef-check job: verify binding parity, README/docs/version freshness
Upgrade prek-action v1 → v2
Update all path triggers: tools/e2e-generator/** → fixtures/** + alef.toml
Update all crate path references to new names
Pin alef install to --tag v0.4.2

Pre-commit:

Add official alef hooks (alef-verify, alef-sync-versions)
Fix clang-format/cppcheck targets for renamed FFI crate

Taskfile:

e2e:generate:* → alef e2e generate
generate-readme → alef readme
version:sync → alef sync-versions
docs:generate:api → alef docs

Test plan

cargo check --workspace passes (excluding wasm target)
cargo test -p tree-sitter-language-pack — 28/28 pass
Rust e2e tests compile and pass (162 tests)
alef e2e validate — all 393 fixtures valid
alef docs generates 14 API reference files
alef readme generates 10 package READMEs
alef sync-versions runs successfully
No stale references to deleted tools/crates

Add changelog entry for #102 (Go bindings and docs fixes). Sync version 1.6.3 across all package manifests. Update dependencies across all lockfiles.

Add missing trait implementations across all public types in ts-pack-core to support alef-generated binding code which requires these traits for FFI serialization, struct construction, and comparison operations. Changes by file: - intel/types.rs: 18 types updated (Span, StructureKind, StructureItem, CommentKind, CommentInfo, DocstringFormat, DocstringInfo, DocSection, ImportInfo, ExportKind, ExportInfo, SymbolKind, SymbolInfo, DiagnosticSeverity, Diagnostic, CodeChunk, ChunkContext, FileMetrics) - extract.rs: 9 types get Default, CaptureOutput gets Hash - node.rs: NodeInfo gets Default + Hash - query.rs: QueryMatch gets Default + PartialEq + Eq

- Delete all hand-written binding crates (python, node, ruby, php, java, wasm, elixir, ffi) and packages (go, csharp, php, python) - Delete custom e2e-generator tool (replaced by alef e2e) - Convert 377 fixture files from flat-assertion format to alef format - Add alef.toml config with 10 target languages, 9 named e2e calls - Add fixture conversion script (scripts/convert_fixtures.py) - Run alef generate + scaffold to produce new binding crates - Update Cargo.toml workspace members and excludes Generated binding crates have compile errors due to opaque type handling in alef (Language, Parser, Tree, DownloadManager). These will be resolved in follow-up commits with further alef upstream fixes.

- Update opaque type paths in alef.toml to use re-export paths - Add ValidationResult, PatternValidation, MatchResult, PatternResult to include types list - Add serde derives to QueryMatch (behind feature flag) - Fix DownloadManager bindings to use Arc<Mutex<T>> for mutable methods - Fix &[&str] parameter handling with Vec<&str> conversion - Fix &ProcessConfig reference handling in PHP/WASM bindings - Fix Vec<_> type annotation in FFI deserialization All binding crates (ffi, py, node, wasm, php) now compile cleanly. Core tests pass.

- Change DownloadManager to use interior mutability (Mutex<Option<T>>) so all methods are &self, enabling Arc<T> wrappers in bindings - Add serde derives to QueryMatch behind feature flag - Add ValidationResult, PatternValidation, MatchResult, PatternResult to alef.toml include types - Update opaque type paths to use core re-export paths - Regenerate all bindings with fixed alef codegen All binding crates (ffi, py, node, wasm, php) compile without errors. Core tests pass (28/28).

These functions have tuple return types that alef's PHP extraction maps incorrectly. Apply manual fixes until extraction handles tuples.

Add fixtures for previously untested API functions: - query/: highlights, injections, locals, run_query (5 fixtures) - tree-inspection/: tree_to_sexp, tree_has_error_nodes (3 fixtures) - extraction/: extract_patterns, validate_extraction (4 fixtures) - registry/: language_count (1 fixture) - language-detection/: extension_ambiguity (3 fixtures) Add alef.toml call definitions for: injections, locals, language_count, extract, validate. All 393 fixtures pass schema validation. Coverage: 25/25 public functions now have at least 1 fixture.

- Update 306 smoke fixtures: root_child_count field → method_result - Change parse call source arg type to "bytes" for Rust .as_bytes() - Skip fixtures for Rust that use unsupported patterns (ambiguity tuples, HashMap field access, non-Result returns, extraction config) - Skip process fixtures using computed fields (import_sources, structure_names) - All Rust e2e tests now compile with 0 errors

- Move input.language into input.config for process fixtures (31 files) - Add comments: true to python_comments fixture - Remove optional flag from default call config arg - Remove download from crate features (default feature, not needed for e2e) - Add ts-pack-node to workspace excludes (no Cargo.toml) - Regenerate e2e tests Rust e2e: 162 tests pass, 0 failures, 2 skipped. Run with: TSLP_LANGUAGES=python,... TSLP_LINK_MODE=static cargo test

- Remove scripts/generate_readme.py, readme_config.yaml, readme_templates/ (replaced by alef readme) - Remove scripts/sync_versions.py (replaced by alef sync-versions) - Remove scripts/convert_fixtures.py (one-time migration, already used) - Remove docs/api/ hand-written API reference (12 files) - Add docs/reference/ generated by alef docs (13 files: per-language API, types, errors, configuration) Kept: tools/snippet-runner, scripts/generate_grammar_table.py, scripts/check_grammar_updates.py, and other grammar management scripts.

- Replace docs/api/ (hand-written) with docs/reference/ (alef-generated) - Update zensical.toml nav to reference/api-*.md paths - Add Rust API reference (now generated by alef) - Add Types, Errors, Configuration reference pages - Align zensical.toml with kreuzberg (material theme, search features) - Fix PHP tuple return type patches (extension_ambiguity, split_code) - Fix QueryMatch captures tuple conversion in Py/Node/PHP bindings

- Add [readme] config to alef.toml (snippets_dir, banner, discord) - Add official alef pre-commit hooks (alef-verify, alef-sync-versions) - Update Taskfile: replace old e2e-generator tasks with alef e2e generate, add docs:generate:api, update generate-readme to use alef readme - Update .task/version.yml to use alef sync-versions - Generate 10 package READMEs via alef readme

- Add alef-check job: verify binding API parity, README freshness, API docs freshness, version sync - Split readme-check into alef-check (alef verify/readme/docs/sync) and snippet-check (snippet-runner) - Upgrade prek-action v1 → v2 - Replace tools/e2e-generator/** path triggers with fixtures/** and alef.toml across all 12 language CI workflows - Remove old generate_readme.py CI reference

- Add crates/tree-sitter-language-pack-{ffi,wasm} to workspace excludes (README-only stub dirs, no Cargo.toml) - Restore tools/snippet-runner in workspace members - Update FFI lib name: ts_pack_ffi → ts_pack_core_ffi in Go binding.go, C# NativeMethods.cs, Java NativeLib.java - Update all .task/*.yml: e2e-generator → alef e2e generate, ts-pack-ffi → ts-pack-core-ffi, old crate paths → new paths - Update all CI workflows: old crate paths → new paths, old crate names → new names, e2e-generator → alef e2e generate - Fix ci-validate.yaml: Ruby/Elixir working dirs to packages/*, pin alef install to --tag v0.4.2 - Fix .pre-commit-config.yaml: clang-format/cppcheck target crates/ts-pack-core-ffi/ Workspace compiles clean. Core tests pass (28/28).

New fixtures for previously untested functions: - registry/: get_language (valid + error), get_parser (valid + error) - error-handling/: process_unknown_language, parse_empty_language - extraction/: extract_invalid_byte_range, validate_empty_patterns - tree-inspection/: split_code_python, tree_error_count_multiple, find_nodes_no_match, root_node_info_javascript - query/: injections_unknown, locals_unknown - process/: javascript_exports_count, python_symbols, python_docstrings, python_all_features Also fixes CI failures: - Add build-system to root pyproject.toml (maturin needs it) - Add crates/ts-pack-core-node/package.json (napi build needs it) - Fix WASM captures tuple .to_string() → format! - Add get_language/get_parser call definitions to alef.toml 411 fixtures total, all pass schema validation.

- Regenerate all e2e test suites from 411 fixtures - Fix python_docstrings fixture (docstring detection not available) - Fix python_all_features fixture (replace docstring check with metrics) - Fix find_nodes_no_match fixture (skip rust, Vec equals unsupported) - Skip query not_empty fixtures for rust (queries not bundled in limited builds) Rust e2e: 44 non-smoke tests pass, 0 failures.

Root pyproject.toml is for dev dependencies (ruff, mypy), not for building. Adding build-system = maturin caused maturin to try parsing the workspace Cargo.toml as a package, failing with "missing field package". The build-system belongs only in packages/python/pyproject.toml.

- Add root pyproject.toml build-system (hatchling, not maturin) to prevent maturin from parsing root pyproject as a maturin project - Add packages/ruby/Gemfile (needed by bundle install in CI) - Fix crates/ts-pack-node → crates/ts-pack-core-node in ci-node.yaml - Fix crates/ts-pack-php → crates/ts-pack-core-php in ci-php.yaml - Fix crates/ts-pack-wasm → crates/ts-pack-core-wasm in ci-wasm.yaml - Fix crates/ts-pack-java → packages/java in publish.yaml

kh3rld

7 CI jobs are currently failing (Rust Tests, Rust E2E, C FFI, Build WASM x2, All Grammars Integration, Lint & Format). Do not merge until CI is green. Inline comments below identify the specific code bugs that need to be fixed.

kh3rld · 2026-04-21T08:22:52Z


 [workspace.package]
-version = "1.6.2"
+version = "1.6.3"


Four new public types (ValidationResult, PatternValidation, MatchResult, PatternResult) and 30+ new trait impls are additions, not patches. By semver this should be 1.7.0 not 1.6.3.

kh3rld · 2026-04-21T08:22:52Z

+app_name = "tree_sitter_language_pack"
+
+[wasm]
+env_shims = ["iswspace", "iswalnum", "towupper", "iswalpha"]


iswlower and towlower are commonly needed wchar functions not listed here. Their absence is a likely cause of the WASM build failure. Audit grammar usage and add any missing shims.

kh3rld · 2026-04-21T08:22:52Z

+
+[[sync.text_replacements]]
+path = "tests/test_apps/java/pom.xml"
+search = "<version>{version}</version>"


Bug: search and replace are identical strings, making the Java version sync a no-op. alef sync-versions will silently skip pom.xml. Fix:

search = '<version>[^<]*</version>' replace = '<version>{version}</version>'

kh3rld · 2026-04-21T08:22:53Z

-        if self.manifest.is_none() {
-            self.manifest = Some(self.fetch_manifest()?);
+        {
+            let mut guard = self.manifest.lock().unwrap();


Use map_err(|e| Error::LockPoisoned(e.to_string()))? instead of .unwrap(). A panic inside fetch_manifest_inner will poison this mutex and make every future caller panic. query.rs already uses the correct pattern. Same issue at lines 128 and 173.

kh3rld · 2026-04-21T08:22:53Z

        }

-        let manifest = self.manifest.as_ref().expect("manifest loaded above");
+        let guard = self.manifest.lock().unwrap();


This guard is held through the download_bundle call (~line 152), which makes a blocking HTTP request. This serializes all concurrent callers for the entire download, defeating the purpose of the &self refactor. Clone bundle.url and bundle.sha256 inside a short lock scope, drop the guard, then call download_bundle.

kh3rld · 2026-04-21T08:22:53Z

-            self.manifest = Some(self.fetch_manifest()?);
+    pub fn ensure_group(&self, group: &str) -> Result<(), Error> {
+        {
+            let mut guard = self.manifest.lock().unwrap();


Same poison risk as line 122. Replace .unwrap() with map_err(|e| Error::LockPoisoned(e.to_string()))?.

kh3rld · 2026-04-21T08:22:53Z


 /// A structural item (function, class, struct, etc.) in source code.
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Default, PartialEq, Eq)]


StructureItem has Eq but not Hash. DocstringInfo (line 162), SymbolInfo (line 237), and Diagnostic (line 263) have the same gap. These types cannot be used as HashMap keys or in a HashSet despite satisfying Eq. Add Hash to all four or document why it is intentionally omitted.

kh3rld · 2026-04-21T08:22:53Z

+crate-type = ["cdylib"]
+
+[dependencies]
+pyo3 = { version = "0.28", features = ["extension-module"] }


Missing abi3-py310 feature. The workspace Cargo.toml declares it but this crate overrides pyo3 directly without inheriting the workspace dependency. Without it the wheel is Python-version-specific (cp310, cp311, etc.) instead of a universal abi3 wheel. Either add "abi3-py310" to features or switch to pyo3 = { workspace = true }.

kh3rld · 2026-04-21T08:22:53Z

+
+[dependencies]
+serde_json = "1"
+tokio = { version = "1", features = ["full"] }


features = ["full"] pulls in the complete Tokio runtime in a C FFI crate. If only the async task wrapper is needed, use ["rt", "rt-multi-thread", "macros"] to keep binary size and compile times down.

kh3rld · 2026-04-21T08:22:53Z

+        run: |
+          alef sync-versions
+          git diff --exit-code -- packages/ crates/ || {
+            echo "::error::Versions are out of sync. Run 'alef sync-versions' and commit."


Add a fixture validation step after this:

- name: Validate e2e fixtures run: alef e2e validate

Without it, stale or malformed fixture YAML files will not fail this job.

Replace 12 separate ci-*.yaml files with one consolidated ci.yaml following the kreuzcrawl pattern: - Add dorny/paths-filter change detection (13 language outputs) - All binding jobs conditional on relevant file changes - Staged dependency graph: validate → rust-tests → build → test - FFI artifact sharing for Go/Java/C#/C - Alef validation: verify, readme, docs, sync-versions freshness - Upgrade prek-action v1 → v2 - Standard env: TSLP_LANGUAGES, TSLP_LINK_MODE, PROJECT_ROOT Kept separate: ci-all-grammars.yaml (2hr+ weekly job), ci-cli.yaml, ci-docker.yaml Deleted: ci-c.yaml, ci-csharp.yaml, ci-elixir.yaml, ci-go.yaml, ci-java.yaml, ci-node.yaml, ci-php.yaml, ci-python.yaml, ci-ruby.yaml, ci-rust.yaml, ci-validate.yaml, ci-wasm.yaml

Goldziher added 17 commits April 20, 2026 10:56

chore: bump to v1.6.3, update changelog and dependencies

fc6f20d

Add changelog entry for #102 (Go bindings and docs fixes). Sync version 1.6.3 across all package manifests. Update dependencies across all lockfiles.

fix(php): patch extension_ambiguity and split_code return types

d3465ba

These functions have tuple return types that alef's PHP extraction maps incorrectly. Apply manual fixes until extraction handles tuples.

chore: add issue templates and CODEOWNERS

87787d6

chore: add PR/issue validation workflows and PR template

9437801

chore: regenerate e2e test suites

c98c6fb

Goldziher requested a review from kh3rld April 21, 2026 07:18

Goldziher added 4 commits April 21, 2026 09:37

kh3rld requested changes Apr 21, 2026

View reviewed changes

Goldziher added 2 commits April 21, 2026 12:13

ci: remove ci-all-grammars.yaml (superseded by consolidated ci.yaml)

f7fa686

Goldziher merged commit f7fa686 into main Apr 21, 2026
5 of 7 checks passed

Goldziher deleted the feat/migrate-to-alef branch April 21, 2026 10:33

Goldziher temporarily deployed to github-pages April 21, 2026 10:39 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: migrate to alef polyglot binding generator#105

feat: migrate to alef polyglot binding generator#105
Goldziher merged 23 commits intomainfrom
feat/migrate-to-alef

Goldziher commented Apr 21, 2026

Uh oh!

kh3rld left a comment

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

kh3rld Apr 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Goldziher commented Apr 21, 2026

Summary

What changed

Test plan

Uh oh!

kh3rld left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants