ci: run the vgi integration suite on every push + expression-filter pushdown

Add a GitHub Actions workflow that runs the canonical Query-farm/vgi
integration sqllogictest suite against the Java example worker on every
push/PR, without building the C++ extension from source. It drives a prebuilt
standalone haybarn-unittest and installs the signed vgi extension via
`INSTALL vgi FROM community` (deps from core); ci/preprocess-require.awk
rewrites each `require <ext>` gate into an explicit INSTALL+LOAD so the
standalone runner can run the tests. ci/run-integration.sh + ci/wrappers/ are
the committed, path-relative replacements for the old /tmp harness. Validated
at 172 pass / 13 skip / 0 fail (skips are the http/bearer/dynamic/
schema_reconcile require-env gates).

Greening the suite surfaced a real gap: spatial_filter_example and
expression_filter_test were zero-row stubs, and table/expression_filter.test
had always silently skipped (the dev harness can't load spatial). Implement
the expression-filter pushdown subsystem end-to-end:

- core: PushdownFilterType.EXPRESSION + PushdownFilter.Expression; decoder
  renders the pushed expression tree to SQL (geoarrow.wkb -> ST_GeomFromHEXWKB),
  mirroring vgi-python's ExpressionNode.to_sql; expressionPredicates() accessors.
- worker: ExpressionFilterEvaluator evaluates predicates against each batch via
  an embedded haybarn_jdbc engine (spatial loaded), bridged in over the Arrow C
  Data interface. haybarn_jdbc is a worker-only dependency; vgi core stays
  engine-free.
- FunctionMetadata.withSupportedExpressionFilters -> FunctionInfo wire field.
- real SpatialFilterExampleFunction + ExpressionFilterTestFunction fixtures
  replace the stubs. expression_filter.test now runs and passes (30 assertions,
  incl. the pushdown EXPLAIN asserts).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: rustyconover

.github/workflows/integration.yml

@@ -0,0 +1,87 @@
+# Copyright 2026 Query Farm LLC - https://query.farm
+#
+# Runs the canonical vgi integration sqllogictest suite against the Java
+# example worker on every push / PR. Rather than building the C++ extension
+# from source, it drives a prebuilt standalone `haybarn-unittest` (the
+# DuckDB/Haybarn sqllogictest runner) and installs the SIGNED vgi extension
+# from the Haybarn community channel + its deps from core. The .test files
+# come from a pinned Query-farm/vgi checkout; ci/preprocess-require.awk turns
+# each `require <ext>` gate into an explicit INSTALL+LOAD so the standalone
+# runner (which links none of these extensions) can run them.
+#
+# See ci/README.md for the full design and the version-pin coupling.
+name: Integration suite
+
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: integration-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  # The Query-farm/vgi commit whose test/sql/integration suite we run. Pinned
+  # for reproducibility; bump deliberately and re-validate against the then-current
+  # community extension (see ci/README.md).
+  VGI_REF: 4444d669c700d9efd70e5f97db4d2c4c886f509f
+  # The Haybarn release providing the prebuilt haybarn-unittest binary. Must be
+  # ABI-compatible with the community-published vgi extension (both v1.5.3).
+  HAYBARN_RELEASE: haybarn-v1.5.3-rc7
+
+jobs:
+  integration:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout vgi-java
+        uses: actions/checkout@v4
+
+      - name: Checkout pinned vgi test suite
+        uses: actions/checkout@v4
+        with:
+          repository: Query-farm/vgi
+          ref: ${{ env.VGI_REF }}
+          path: vgi-upstream
+
+      - name: Set up JDK 25
+        uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '25'
+
+      - name: Cache Gradle
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.gradle/caches
+            ~/.gradle/wrapper
+          key: gradle-${{ runner.os }}-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
+          restore-keys: gradle-${{ runner.os }}-
+
+      - name: Build example worker
+        run: ./gradlew --no-daemon :vgi-example-worker:installDist
+
+      - name: Download haybarn-unittest
+        run: |
+          gh release download "$HAYBARN_RELEASE" \
+            --repo Query-farm-haybarn/haybarn \
+            --pattern 'haybarn_unittest-linux-amd64.zip' \
+            --output /tmp/haybarn-unittest.zip --clobber
+          unzip -o -q /tmp/haybarn-unittest.zip -d /tmp/haybarn-unittest
+          UNITTEST=$(find /tmp/haybarn-unittest -name 'haybarn-unittest' -type f | head -1)
+          chmod +x "$UNITTEST"
+          echo "HAYBARN_UNITTEST=$UNITTEST" >> "$GITHUB_ENV"
+        env:
+          GH_TOKEN: ${{ github.token }}
+
+      - name: Run integration suite
+        run: ci/run-integration.sh
+        env:
+          VGI_SRC: ${{ github.workspace }}/vgi-upstream
+          VGI_WORKER_BIN: ${{ github.workspace }}/vgi-example-worker/build/install/vgi-example-worker/bin/vgi-example-worker
+          # JDK 25 + native access for the worker JVM (FFM shm, haybarn_jdbc).
+          JAVA_TOOL_OPTIONS: --enable-native-access=ALL-UNNAMED

.gitignore

@@ -6,3 +6,7 @@ build/
 .vscode/
 .DS_Store
 /tmp/
+# Local scratch / test artifacts
+duckdb_unittest_tempdir/
+.wrangler/
+.playwright-mcp/

CLAUDE.md

@@ -248,6 +248,63 @@ older interfaces** (`TableFunction`, `TableInOutFunction`, etc.) — the
 `ScalarFn` style hasn't been extended to those because their richer
 lifecycle methods + per-execution state don't translate one-for-one.
 
+## State of play (as of 2026-06-12)
+
+**2026-06-12 — GitHub Actions integration CI + expression-filter pushdown.**
+Two coupled pieces landed so the integration suite runs on every push/PR
+without a C++ build:
+
+- **CI (`.github/workflows/integration.yml` + `ci/`).** Drives the **prebuilt**
+  standalone `haybarn-unittest` (Haybarn release asset) against the Java worker,
+  installing the **signed** vgi extension via `INSTALL vgi FROM community` (deps
+  from `core`) — no extension build from source. The `.test` files come from a
+  pinned `Query-farm/vgi` checkout (`VGI_REF` in the workflow `env:`). The
+  standalone runner links **none** of vgi/httpfs/json/parquet/spatial, so
+  `ci/preprocess-require.awk` rewrites every `require <ext>` into an explicit
+  `INSTALL … FROM {community,core}; LOAD …;` (`require-env` untouched).
+  `ci/run-integration.sh` stages the preprocessed tree, sets the four
+  `VGI_*_WORKER` vars (`launch:` + the three `ci/wrappers/` catalog wrappers —
+  the committed, path-relative replacements for the old `/tmp/vgi-worker-*`),
+  warms the extension cache once, and tallies pass/skip/fail. Validated locally
+  at **172 pass / 13 skip / 0 fail** (the 13 skips are the http/bearer/dynamic/
+  `schema_reconcile` `require-env` gates). **Path-B coupling:** the extension is
+  pulled live from community (not version-pinned), so a `VGI_REF` bump must be
+  re-validated against the then-current community build — see `ci/README.md`.
+
+- **Expression-filter pushdown subsystem** (was entirely absent; this is why
+  `spatial_filter_example`/`expression_filter_test` were zero-row `Stub`s and
+  `table/expression_filter.test` always *skipped* — the local dev harness can't
+  load spatial). Now implemented end-to-end:
+  - **Wire decode (core):** `PushdownFilterType.EXPRESSION` +
+    `PushdownFilter.Expression(columnName, columnIndex, sql)`.
+    `PushdownFiltersDecoder` renders the pushed expression tree
+    (`column_ref`/`constant`/`function`/`comparison`/`conjunction`, constants
+    resolved from sibling `value_ref` columns, geoarrow.wkb →
+    `ST_GeomFromHEXWKB`) to a SQL predicate — mirroring vgi-python's
+    `ExpressionNode.to_sql`. `evaluate()` skips Expression (not row-at-a-time);
+    `PushdownFilters.expressionPredicates()` / `FilterApplier.expressionPredicates()`
+    expose the SQL.
+  - **Evaluator (worker):** `example/table/ExpressionFilterEvaluator` evaluates
+    the predicates against each batch via an embedded **haybarn_jdbc** engine
+    (Maven Central `farm.query.haybarn:haybarn_jdbc:1.5.3` + `arrow-c-data`),
+    thread-local connection with spatial loaded — the Java analogue of
+    vgi-python's `vgi._duckdb` evaluator. Batch is bridged in via the Arrow C
+    Data interface (`registerArrowStream`); a registered stream is **single-use**
+    so it's queried once per batch; `FilterApplier.compact` does the masking.
+    **haybarn_jdbc is a worker-only dep — the published `vgi` core stays
+    engine-free.**
+  - **Capability wiring:** `FunctionMetadata.withSupportedExpressionFilters(...)`
+    → `FunctionInfo.supported_expression_filters` (was hardcoded `List.of()` at
+    `VgiServiceImpl` ~line 1731). The C++ extension only pushes an expression
+    filter when every function name in the tree is declared; otherwise it keeps
+    a FILTER node (the `length(name) > 7` EXPLAIN case).
+  - **Fixtures:** real `SpatialFilterExampleFunction` (`{n,x,y,geom GEOMETRY}`
+    grid, declares `&&`/`st_intersects_extent`) + `ExpressionFilterTestFunction`
+    (`{id,name,tags,score}`, declares `list_contains`/`starts_with`/`contains`)
+    replace the two stubs. `table/expression_filter.test` now **runs and passes**
+    (30 assertions, incl. the pushdown EXPLAIN asserts) — strictly better than
+    the prior silent skip.
+
 ## State of play (as of 2026-06-11)
 
 **2026-06-11 — ported the evolved vgi-python storage layer** (counters + ranged

ci/README.md

@@ -0,0 +1,67 @@
+# CI: the vgi integration suite
+
+[`.github/workflows/integration.yml`](../.github/workflows/integration.yml)
+runs the canonical [Query-farm/vgi](https://github.com/Query-farm/vgi)
+integration sqllogictest suite against this repo's Java example worker on every
+push / PR. The same `.test` files run against the Python and Go ports, so a
+green run here is real wire-compatibility evidence.
+
+## How it works (no C++ build)
+
+Rather than building the vgi DuckDB extension from source, CI drives a
+**prebuilt** standalone `haybarn-unittest` (the DuckDB/Haybarn sqllogictest
+runner, published in Haybarn's releases) and installs the **signed** vgi
+extension from the Haybarn community channel:
+
+1. **Build the worker** — `./gradlew :vgi-example-worker:installDist`.
+2. **Checkout the test suite** — `Query-farm/vgi` at a pinned commit; its
+   `test/sql/integration/*.test` files are the suite.
+3. **Download the runner** — `haybarn_unittest-linux-amd64.zip` from the pinned
+   Haybarn release.
+4. **Preprocess** — the standalone runner links none of the extensions the
+   tests gate on, so [`preprocess-require.awk`](preprocess-require.awk) rewrites
+   each `require <ext>` into an explicit signed `INSTALL <ext> FROM
+   {community,core}; LOAD <ext>;`. Everything else (notably `require-env`) is
+   left untouched.
+5. **Run** — [`run-integration.sh`](run-integration.sh) stages the preprocessed
+   tree, points the four `VGI_*_WORKER` env vars at the worker (via `launch:`,
+   which amortises JVM cold-start across the run) and the three catalog
+   [`wrappers/`](wrappers), warms the extension cache once, then runs each
+   `.test` and tallies pass / skip / fail.
+
+Out of scope and excluded: `writable/`, `simple_writable/` (the port is
+read-only), and `nested_type_combinations.test` (segfaults the upstream
+runner — see the project `CLAUDE.md`). The HTTP / bearer / dynamic-code /
+`schema_reconcile` tests skip via their `require-env` gates (we don't set those
+workers), exactly as in the reference harness.
+
+## Run it locally
+
+```bash
+./gradlew :vgi-example-worker:installDist
+VGI_SRC=~/path/to/vgi-checkout \
+HAYBARN_UNITTEST=/path/to/haybarn-unittest \
+VGI_WORKER_BIN="$PWD/vgi-example-worker/build/install/vgi-example-worker/bin/vgi-example-worker" \
+  ci/run-integration.sh
+```
+
+## Version pins (and their coupling)
+
+Two pins live in the workflow's `env:` block:
+
+| Pin | What | Why |
+|-----|------|-----|
+| `VGI_REF` | the `Query-farm/vgi` commit supplying the `.test` files | reproducibility — bump deliberately |
+| `HAYBARN_RELEASE` | the Haybarn release supplying `haybarn-unittest` | must be ABI-compatible with the community vgi extension (both `v1.5.3`) |
+
+**The coupling to know about:** the vgi extension is pulled live from the
+community channel (`INSTALL vgi FROM community`), which always serves the
+*currently published* build — it is not version-pinned here. So CI verifies the
+Java worker against **what users can actually install today**. If `VGI_REF`
+points at a commit whose tests exercise a protocol feature the published
+extension doesn't yet ship (or vice-versa), that test can fail or skip. When
+bumping `VGI_REF`, re-run the suite locally against the current community
+extension and adjust the pin / wrappers together. The catalog
+[`wrappers/`](wrappers) encode the canonical `versioned` /
+`versioned_tables` / `attach_options` fixture version sets and must track those
+fixtures when they move.

ci/preprocess-require.awk

@@ -0,0 +1,18 @@
+# Copyright 2026 Query Farm LLC - https://query.farm
+#
+# Rewrite each `require <ext>` gate in an upstream vgi sqllogictest into an
+# explicit signed INSTALL+LOAD, so the prebuilt standalone `haybarn-unittest`
+# (which links none of these extensions) can run the suite. The vgi extension
+# comes from the signed community channel; httpfs/json/parquet/spatial from the
+# signed core channel. `require-env` and every other directive pass through
+# untouched. See ci/README.md.
+/^require[ \t]+vgi[ \t]*$/ {
+    print "statement ok"; print "INSTALL vgi FROM community;"; print "";
+    print "statement ok"; print "LOAD vgi;"; next
+}
+/^require[ \t]+(httpfs|json|parquet|spatial)[ \t]*$/ {
+    ext = $2
+    print "statement ok"; print "INSTALL " ext " FROM core;"; print "";
+    print "statement ok"; print "LOAD " ext ";"; next
+}
+{ print }

ci/run-integration.sh

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+# Copyright 2026 Query Farm LLC - https://query.farm
+#
+# Run the vgi integration sqllogictest suite against the Java example worker,
+# using a prebuilt standalone `haybarn-unittest` and the signed community vgi
+# extension (no C++ build from source). See ci/README.md for the approach.
+#
+# Required environment:
+#   VGI_SRC          path to a Query-farm/vgi checkout (contains test/sql/integration)
+#   HAYBARN_UNITTEST path to the haybarn-unittest binary
+#   VGI_WORKER_BIN   path to the built example worker launcher
+#                    (vgi-example-worker/build/install/.../bin/vgi-example-worker)
+# Optional:
+#   STAGE            scratch dir for the preprocessed test tree (default: mktemp)
+set -euo pipefail
+
+: "${VGI_SRC:?path to a Query-farm/vgi checkout}"
+: "${HAYBARN_UNITTEST:?path to the haybarn-unittest binary}"
+: "${VGI_WORKER_BIN:?path to the example worker launcher}"
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+STAGE="${STAGE:-$(mktemp -d)}"
+INTEGRATION="$VGI_SRC/test/sql/integration"
+[ -d "$INTEGRATION" ] || { echo "::error::no test/sql/integration under VGI_SRC=$VGI_SRC"; exit 1; }
+
+echo "Staging preprocessed tests into $STAGE ..."
+mkdir -p "$STAGE/test/sql/integration"
+( cd "$INTEGRATION"
+  # writable/simple_writable are out of scope (read-only port);
+  # nested_type_combinations segfaults the upstream runner (documented in CLAUDE.md).
+  find . -name '*.test' \
+       -not -path '*/writable/*' -not -path '*/simple_writable/*' \
+       -not -name 'nested_type_combinations.test' | while read -r f; do
+    mkdir -p "$STAGE/test/sql/integration/$(dirname "$f")"
+    awk -f "$HERE/preprocess-require.awk" "$f" > "$STAGE/test/sql/integration/$f"
+  done )
+
+# launch: amortises the JVM cold-start across the whole run via a flock-coordinated
+# AF_UNIX worker pool. VGI_TEST_DEDICATED_WORKER is a plain (non-pooled) worker for
+# the crash/pool-recovery tests. The three wrappers route the one binary into the
+# versioned / versioned_tables / attach_options catalogs.
+export VGI_WORKER_BIN
+export VGI_TEST_WORKER="launch:${VGI_WORKER_BIN}"
+export VGI_TEST_DEDICATED_WORKER="${VGI_WORKER_BIN}"
+export VGI_VERSIONED_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned"
+export VGI_VERSIONED_TABLES_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned-tables"
+export VGI_ATTACH_OPTIONS_WORKER="launch:${HERE}/wrappers/vgi-worker-attach-options"
+
+cd "$STAGE"
+
+echo "Warming the extension cache (vgi from community, deps from core) ..."
+mkdir -p "$STAGE/test"
+cat > "$STAGE/test/_warm.test" <<'EOF'
+# name: test/_warm.test
+# group: [warm]
+statement ok
+INSTALL vgi FROM community;
+
+statement ok
+INSTALL httpfs FROM core;
+
+statement ok
+INSTALL json FROM core;
+
+statement ok
+INSTALL parquet FROM core;
+
+statement ok
+INSTALL spatial FROM core;
+EOF
+"$HAYBARN_UNITTEST" "test/_warm.test" >/dev/null 2>&1 || echo "::warning::extension warm step did not fully succeed"
+rm -f "$STAGE/test/_warm.test"
+
+echo "Running suite ..."
+pass=0; skip=0; fail=0; failed=()
+while IFS= read -r t; do
+  rel="${t#"$STAGE"/}"
+  if out=$("$HAYBARN_UNITTEST" "$rel" 2>&1); then :; fi
+  if grep -q "All tests passed" <<<"$out"; then
+    pass=$((pass + 1))
+  elif grep -qE "All tests were skipped|No tests ran" <<<"$out"; then
+    skip=$((skip + 1))
+  else
+    fail=$((fail + 1)); failed+=("$rel")
+    echo "----- FAIL: $rel -----"
+    printf '%s\n' "$out" | tail -40
+  fi
+done < <(find "$STAGE/test" -name '*.test' | sort)
+
+echo "=================================================="
+echo "PASS=$pass  SKIP=$skip  FAIL=$fail"
+if [ "$fail" -gt 0 ]; then
+  printf '  failed: %s\n' "${failed[@]}"
+  exit 1
+fi

ci/wrappers/vgi-worker-attach-options

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+# Routes the single example worker binary into the `attach_options` catalog
+# (Main.java short-circuits this catalog name to the echo_attach_options
+# worker). Worker binary named via $VGI_WORKER_BIN.
+export VGI_WORKER_CATALOG_NAME=attach_options
+exec "${VGI_WORKER_BIN:?set VGI_WORKER_BIN to the example worker launcher}" "$@"

ci/wrappers/vgi-worker-versioned

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Routes the single example worker binary into the canonical `versioned`
+# catalog (vgi-python/_test_fixtures/versioned.py). Must stay in sync with that
+# fixture's version sets. The worker binary is named via $VGI_WORKER_BIN.
+export VGI_WORKER_CATALOG_NAME=versioned
+export VGI_WORKER_IMPLEMENTATION_VERSION=1.0.0
+export VGI_WORKER_SUPPORTED_IMPL_VERSIONS=1.0.0
+export VGI_WORKER_DATA_VERSION_SPEC='>=1.0.0,<2.0.0'
+export VGI_WORKER_SUPPORTED_VERSIONS=1.0.0,1.1.0,1.2.0
+export VGI_WORKER_DEFAULT_DATA_VERSION=1.2.0
+exec "${VGI_WORKER_BIN:?set VGI_WORKER_BIN to the example worker launcher}" "$@"

ci/wrappers/vgi-worker-versioned-tables

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Routes the single example worker binary into the canonical `versioned_tables`
+# catalog (vgi-python/_test_fixtures/versioned_tables.py). Must stay in sync
+# with that fixture's version sets. Worker binary named via $VGI_WORKER_BIN.
+export VGI_WORKER_CATALOG_NAME=versioned_tables
+export VGI_WORKER_IMPLEMENTATION_VERSION=11.0.0
+export VGI_WORKER_SUPPORTED_IMPL_VERSIONS=10.0.0,10.1.0,11.0.0
+export VGI_WORKER_DATA_VERSION_SPEC='>=1.0.0,<4.0.0'
+export VGI_WORKER_SUPPORTED_VERSIONS=1.0.0,1.1.0,2.0.0,3.0.0
+export VGI_WORKER_DEFAULT_DATA_VERSION=3.0.0
+exec "${VGI_WORKER_BIN:?set VGI_WORKER_BIN to the example worker launcher}" "$@"

vgi-example-worker/build.gradle.kts

@@ -6,6 +6,15 @@ plugins {
 dependencies {
     implementation(project(":vgi"))
     implementation("org.slf4j:slf4j-simple:2.0.16")
+    // Embedded Haybarn engine for evaluating pushed expression filters
+    // (spatial &&, list_contains, ...) against emitted batches — mirrors
+    // vgi-python's `vgi._duckdb` expression-filter evaluator. arrow-c-data
+    // bridges an Arrow batch into the engine via the C Data interface.
+    implementation("farm.query.haybarn:haybarn_jdbc:1.5.3")
+    implementation("org.apache.arrow:arrow-c-data:18.1.0")
+
+    testImplementation("org.junit.jupiter:junit-jupiter:5.11.3")
+    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
 }
 
 application {