fix(#4382): Improve handling of existing tables in PostgreSQL sink

[stefaniesinner]

Purpose

This continues #4383 and fixes #4382 by improving the PostgreSQL sink in three ways:

• Adds an append mode for existing tables and batch inserts
• Only checks the table in the current schema, so same-name tables in other schemas no longer break validation
• Lists all missing columns, or all type mismatches with readable type names, in one error message

A v0 to v1 migration keeps existing pipelines unchanged.

I added tests (validation, migration, batching) and updated the PostgreSQL documentation with the new options.

Results

Configuring the sink now shows a toggle for writing into an existing table and a batch size field. When enabled, the sink writes into the existing table instead of creating one, and events are inserted in batches of the given size.

Remarks

PR introduces (a) breaking change(s): no

PR introduces (a) deprecation(s): no

[stefaniesinner]

The UI shows the new toggle's description twice. Also, it would probably be better to move the two new fields (e.g. below the table name) instead of leaving them at the bottom. I left both out here, since they are separate UI concerns.

[Copilot]

Pull request overview

This PR improves the PostgreSQL sink in streampipes-sinks-databases-jvm to better support writing into pre-existing tables by adding an “append” mode, batch inserts, schema-scoped table introspection, and more informative table validation errors. It also introduces a v0→v1 model migration to keep existing pipelines compatible and adds/updates tests and documentation.

Changes:

• Add sink configuration options for “Use Existing Table” and “Batch Size”, including a v0→v1 migration.
• Restrict PostgreSQL schema introspection to the current schema and improve validation error messages (missing columns/type mismatches).
• Add JDBC batching support and tests covering migration, validation, and batching behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/DatabaseSinksExtensionModuleExport.java	Registers the new PostgreSQL sink migrator.
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/jdbcclient/JdbcClient.java	Adds append-mode behavior when table is missing and introduces batch insert execution/flush logic.
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/jdbcclient/model/StatementHandler.java	Adds batch buffering/execution APIs used by JdbcClient.
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/jdbcclient/model/TableDescription.java	Improves validation to report all missing columns or all type mismatches with readable type names.
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/postgresql/PostgreSql.java	Applies append/batch parameters and restricts introspection to current_schema().
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/postgresql/PostgreSqlParameters.java	Adds append/batch parameters and sane defaults.
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/postgresql/PostgreSqlSink.java	Bumps model version to 1 and exposes new UI configuration options (toggle + batch size).
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/postgresql/migration/PostgreSqlSinkMigrationV1.java	Introduces the v0→v1 migration adding the two new static properties.
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/resources/org.apache.streampipes.sinks.databases.jvm.postgresql/documentation.md	Documents “Use Existing Table” and “Batch Size”.
streampipes-extensions/streampipes-sinks-databases-jvm/src/main/resources/org.apache.streampipes.sinks.databases.jvm.postgresql/strings.en	Adds localized strings for the two new options.
streampipes-extensions/streampipes-sinks-databases-jvm/src/test/java/org/apache/streampipes/sinks/databases/jvm/jdbcclient/model/StatementHandlerTest.java	Adds tests for batch buffering/execution behavior.
streampipes-extensions/streampipes-sinks-databases-jvm/src/test/java/org/apache/streampipes/sinks/databases/jvm/jdbcclient/model/TableDescriptionTest.java	Adds tests ensuring improved validation messages.
streampipes-extensions/streampipes-sinks-databases-jvm/src/test/java/org/apache/streampipes/sinks/databases/jvm/postgresql/migration/PostgreSqlSinkMigrationV1Test.java	Adds tests for the v0→v1 migration behavior and defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tenthe]

The UI shows the new toggle's description twice. Also, it would probably be better to move the two new fields (e.g. below the table name) instead of leaving them at the bottom. I left both out here, since they are separate UI concerns.

The field order is defined by the order of the properties in declareConfig(): the builder appends them in that order, and the UI renders staticProperties as-is. So if we want to change the order of these new fields, it should be done in this PR.

The duplicated text is a generic slide-toggle rendering issue. sp-app-static-property already shows the description, and StaticSlideToggleComponent renders the same staticProperty.description again inside the mat-slide-toggle. This should be addressed in a separate PR.

[tenthe]

@stefaniesinner thanks for the PR! Appreciate you taking care of this.