Skip to content

Commit ce6e4f9

Browse files
ferhatelmasferhatelmas
committed
fix: add acceptance matrix for vectors
* add orioledb support * harden validation * make pgvector closer to s3vectors * improve lifecycle management Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
1 parent 0afb093 commit ce6e4f9

49 files changed

Lines changed: 6479 additions & 304 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.docker/docker-compose-infra-oriole-pgvector-override.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
services:
2+
tenant_db:
3+
build:
4+
context: .
5+
dockerfile: .docker/orioledb-pgvector.Dockerfile
6+
image: supabase-storage-orioledb-pgvector:pg17-local

‎.docker/orioledb-pgvector.Dockerfile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
ARG ORIOLEDB_IMAGE=orioledb/orioledb:latest-pg17
2+
FROM ${ORIOLEDB_IMAGE}
3+
4+
ARG PGVECTOR_VERSION=0.8.2
5+
6+
USER root
7+
8+
RUN apk add --no-cache --virtual .pgvector-build-deps \
9+
build-base \
10+
clang \
11+
git \
12+
llvm \
13+
&& git clone --depth 1 --branch "v${PGVECTOR_VERSION}" \
14+
https://github.com/pgvector/pgvector.git /tmp/pgvector \
15+
&& cd /tmp/pgvector \
16+
&& make clean \
17+
&& make OPTFLAGS="" \
18+
&& make install \
19+
&& rm -rf /tmp/pgvector \
20+
&& apk del .pgvector-build-deps
21+
22+
USER postgres

‎.env.acceptance.sample‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ ACCEPTANCE_ENABLE_RENDER=true
2424
# ACCEPTANCE_RLS_READ_OBJECT points at an existing object readable by authenticated role.
2525
ACCEPTANCE_ENABLE_RLS_SETUP=true
2626
# ACCEPTANCE_RLS_READ_OBJECT=authenticated/casestudy.png
27-
# Vector acceptance requires a configured S3 Vectors-compatible service.
27+
# Vector acceptance requires either the local pgvector provider or a configured
28+
# S3 Vectors-compatible service.
2829
ACCEPTANCE_ENABLE_VECTOR=false
2930
ACCEPTANCE_ENABLE_ICEBERG=true
3031
ACCEPTANCE_ENABLE_WIRE=true

‎.github/workflows/acceptance.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,14 +81,18 @@ jobs:
8181
env:
8282
ACCEPTANCE_ADMIN_URL: ${{ matrix.tenancy == 'multitenant' && 'http://127.0.0.1:5001' || '' }}
8383
ACCEPTANCE_ENABLE_ADMIN: ${{ matrix.tenancy == 'multitenant' && 'true' || 'false' }}
84-
ACCEPTANCE_INFRA_RESTART_SCRIPT: ${{ matrix.database == 'oriole' && 'infra:restart:ci:oriole' || 'infra:restart:ci' }}
84+
ACCEPTANCE_ENABLE_VECTOR: "true"
85+
ACCEPTANCE_INFRA_RESTART_SCRIPT: ${{ matrix.database == 'oriole' && 'infra:restart:ci:oriole:pgvector' || 'infra:restart:ci' }}
8586
ACCEPTANCE_PROFILE: ${{ inputs.profile || 'full' }}
8687
ACCEPTANCE_X_FORWARDED_HOST: ${{ matrix.tenancy == 'multitenant' && 'bjhaohmqunupljrqypxz.local.dev' || '' }}
8788
MULTI_TENANT: ${{ matrix.tenancy == 'multitenant' && 'true' || 'false' }}
8889
PG_QUEUE_ENABLE: ${{ matrix.tenancy == 'multitenant' && 'true' || 'false' }}
8990
REQUEST_X_FORWARDED_HOST_REGEXP: ${{ matrix.tenancy == 'multitenant' && '^([a-z]{20})[.]local[.](?:com|dev)$' || '' }}
9091
STORAGE_PUBLIC_URL: ${{ matrix.tenancy == 'multitenant' && 'http://127.0.0.1:5000' || '' }}
9192
STORAGE_BACKEND: ${{ matrix.storage_backend }}
93+
VECTOR_BUCKET_PROVIDER: pgvector
94+
VECTOR_DATABASE_URL: ${{ matrix.tenancy == 'single' && 'postgresql://postgres:postgres@127.0.0.1:5432/postgres' || '' }}
95+
VECTOR_STORE_MIGRATIONS_ENABLED: "true"
9296
run: |
9397
mkdir -p data coverage/acceptance
9498
chmod -R 777 data

‎acceptance/API_COVERAGE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ the configured target, and the selected profile includes the spec:
4242
| Render | `ACCEPTANCE_ENABLE_RENDER=true` | public, authenticated, and signed image transformation routes, `webp` output format, non-image input errors, invalid transformation validation |
4343
| RLS | `ACCEPTANCE_ENABLE_RLS_SETUP=true` plus anon/authenticated keys and RLS resource config | authenticated allow and anon deny for read/write on configured policies |
4444
| Path edges | Derived from `ACCEPTANCE_TARGET` and `STORAGE_BACKEND` | list-v2 preservation for object names with empty path segments; local S3/MinIO backends skip this case directly |
45-
| Vector | `ACCEPTANCE_ENABLE_VECTOR=true` | vector bucket pagination, index pagination, put/get/list/query/delete lifecycle |
45+
| Vector | `ACCEPTANCE_ENABLE_VECTOR=true` with local pgvector or S3 Vectors configured | vector bucket pagination, index pagination, vector list pagination, metadata filter keys, non-filterable metadata rejection, default distance omission, cosine and euclidean query behavior, put/get/list/query/delete lifecycle |
4646
| Iceberg | `ACCEPTANCE_ENABLE_ICEBERG=true` | analytics bucket, catalog config, namespace (create/list/load/head/drop, missing load/drop, upsert on re-create, drop blocked when non-empty), table create/list/page-size/load/head/drop, missing load/drop, commit success/conflict |
4747

4848
## Intentionally Gated

‎acceptance/README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,15 @@ Local CI also enables admin acceptance for multitenant matrix entries. Path-edge
9898
from the local storage backend, so empty path segment object names are exercised only on backends
9999
that can store them.
100100

101+
Local CI enables vector acceptance on PostgreSQL and OrioleDB matrix rows using the pgvector-backed
102+
local provider, covering both S3/file storage backends and single/multitenant modes. OrioleDB rows
103+
use the locally built OrioleDB+pgvector image. Single-tenant rows create and migrate a dedicated
104+
`storage_vectors` database from `VECTOR_DATABASE_URL`. Multitenant pgvector rows provision the local
105+
tenant with the configured tenant database URL and pool URL. Multitenant pgvector index DDL reuses
106+
the active tenant transaction connection; single-tenant pgvector and S3 Vectors index creation keep
107+
physical side effects outside retried metadata transactions and clean up committed metadata on
108+
post-commit failures.
109+
101110
## GitHub Environments
102111

103112
The workflow dispatch `acceptance_environment` input uses `local` for the managed local run. Any
@@ -125,7 +134,7 @@ secrets as environment secrets.
125134
| `ACCEPTANCE_ENABLE_CDN` | Enables CDN purge tests. Managed local runs provide a purge stub by default. |
126135
| `ACCEPTANCE_ENABLE_RENDER` | Enables image transformation tests. |
127136
| `ACCEPTANCE_ENABLE_RLS_SETUP` | Enables RLS tests; requires service, anon, authenticated keys and bucket/prefix policy resources. |
128-
| `ACCEPTANCE_ENABLE_VECTOR` | Enables vector bucket API tests. Requires a configured S3 Vectors-compatible service. |
137+
| `ACCEPTANCE_ENABLE_VECTOR` | Enables vector bucket API tests. Requires local pgvector or a configured S3 Vectors target. |
129138
| `ACCEPTANCE_ENABLE_ICEBERG` | Enables Iceberg catalog API tests. |
130139
| `ACCEPTANCE_ENABLE_WIRE` | Enables wire-level tests outside the `wire` / `full` profiles. |
131140
| `ACCEPTANCE_RLS_BUCKET` | Bucket used by opt-in RLS tests. Defaults to local dummy `bucket2`. |

‎acceptance/scripts/run-managed-local.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -277,7 +277,11 @@ function closeHttpServer(server: HttpServer): Promise<void> {
277277

278278
function resolveInfraRestartScript() {
279279
const script = acceptanceEnv('ACCEPTANCE_INFRA_RESTART_SCRIPT') ?? 'infra:restart:ci'
280-
const allowed = new Set(['infra:restart:ci', 'infra:restart:ci:oriole'])
280+
const allowed = new Set([
281+
'infra:restart:ci',
282+
'infra:restart:ci:oriole',
283+
'infra:restart:ci:oriole:pgvector',
284+
])
281285

282286
if (!allowed.has(script)) {
283287
throw new Error(`Unsupported ACCEPTANCE_INFRA_RESTART_SCRIPT: ${script}`)

0 commit comments

Comments
 (0)