Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 31 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
## Features

- Foreign Data Wrapper: `lance_fdw`
- Auto schema discovery + DDL: `lance_import(server, schema, table, uri, batch_size)`
- Auto schema discovery + DDL: `lance_import(server, schema, table, uri, batch_size => NULL)`
- Native-first type mapping:
- Scalars map to native PostgreSQL scalar types where possible
- `list<T>` maps to `T[]` when possible
Expand Down Expand Up @@ -59,7 +59,7 @@ SELECT lance_import(
'public',
'my_lance_table',
'/path/to/your/lance/table',
NULL
batch_size => NULL
);
```

Expand All @@ -68,10 +68,6 @@ SELECT lance_import(
- The foreign table `public.my_lance_table`
- Composite types for nested `struct` fields, e.g. `public.lance_my_lance_table_meta`

Why not `IMPORT FOREIGN SCHEMA`?

- PostgreSQL's `IMPORT FOREIGN SCHEMA` callback can only emit `CREATE FOREIGN TABLE` statements, so it cannot also create composite types for nested `struct` columns. `lance_import` performs both type and table DDL in one step.

### 3) Query like a regular table

```sql
Expand All @@ -80,6 +76,35 @@ SELECT count(*) FROM public.my_lance_table;
SELECT * FROM public.my_lance_table LIMIT 10;
```

### 4) Attach and sync a Lance namespace

```sql
-- Plan only (no DDL).
SELECT *
FROM lance_attach_namespace('lance_srv', dry_run => true);

-- Attach a namespace subtree into local schemas/tables.
SELECT *
FROM lance_attach_namespace(
'lance_srv',
root_namespace_id => ARRAY[]::text[],
schema_prefix => 'lance',
batch_size => NULL,
limit_per_list_call => 1000,
dry_run => false
);

-- Reconcile local objects with the remote namespace.
SELECT *
FROM lance_sync_namespace(
'lance_srv',
schema_prefix => 'lance',
drop_missing => false,
recreate_changed => false,
dry_run => true
);
```

## Type Mapping (native-first)

| Arrow/Lance Type | PostgreSQL Type |
Expand Down
2 changes: 1 addition & 1 deletion tests/sql/00_smoke.slt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Basic smoke test for lance FDW.

statement ok
SELECT lance_import('${SERVER}', '${SCHEMA}', 't_fdw', '${LANCE_URI}', NULL);
SELECT lance_import('${SERVER}', '${SCHEMA}', 't_fdw', '${LANCE_URI}', batch_size => NULL);

query I
SELECT count(*) FROM t_fdw;
Expand Down
9 changes: 7 additions & 2 deletions tests/sql/01_type_mapping.slt
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
# Ensure declared type mapping matches runtime value conversion.

statement ok
SELECT lance_import('${SERVER}', '${SCHEMA}', 't_misc', '${LANCE_URI_MISC}', NULL);
SELECT lance_import(
'${SERVER}',
'${SCHEMA}',
't_misc',
'${LANCE_URI_MISC}',
batch_size => NULL
);

query T
SELECT atttypid::regtype::text
Expand Down Expand Up @@ -34,4 +40,3 @@ query T
SELECT (dec IS NULL) AND (dict IS NULL) FROM t_misc WHERE u16 = 2;
----
t

9 changes: 8 additions & 1 deletion tests/sql/02_namespace_attach.slt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,14 @@ CREATE SERVER ${SERVER} FOREIGN DATA WRAPPER lance_fdw
statement ok
CREATE TEMP TABLE attach_result AS
SELECT local_schema, local_table, status, detail
FROM lance_attach_namespace('${SERVER}', ARRAY[]::text[], '${SCHEMA}', NULL, 1000, false);
FROM lance_attach_namespace(
'${SERVER}',
root_namespace_id => ARRAY[]::text[],
schema_prefix => '${SCHEMA}',
batch_size => NULL,
limit_per_list_call => 1000,
dry_run => false
);

query I
SELECT count(*) FROM attach_result;
Expand Down
19 changes: 16 additions & 3 deletions tests/sql/05_namespace_sync.slt
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,14 @@ CREATE FOREIGN TABLE ${SCHEMA}.ghost(id int4)

query TTT
SELECT local_table, action, status
FROM lance_sync_namespace('${SERVER}', ARRAY[]::text[], '${SCHEMA}', false, false, false)
FROM lance_sync_namespace(
'${SERVER}',
root_namespace_id => ARRAY[]::text[],
schema_prefix => '${SCHEMA}',
drop_missing => false,
recreate_changed => false,
dry_run => false
)
ORDER BY local_table, action, status;
----
ghost drift ok
Expand All @@ -28,7 +35,14 @@ train create_table ok

query TTT
SELECT local_table, action, status
FROM lance_sync_namespace('${SERVER}', ARRAY[]::text[], '${SCHEMA}', true, true, false)
FROM lance_sync_namespace(
'${SERVER}',
root_namespace_id => ARRAY[]::text[],
schema_prefix => '${SCHEMA}',
drop_missing => true,
recreate_changed => true,
dry_run => false
)
ORDER BY local_table, action, status;
----
ghost drop_table ok
Expand All @@ -49,4 +63,3 @@ query T
SELECT to_regclass('${SCHEMA}.ghost') IS NULL;
----
t

Loading