[Bug]: Vertex batch accounting fails when gcsOutputDirectory exceeds 255 characters #7160

Prerequisites

• I have searched existing issues and discussions to avoid duplicates
• I am using the latest version (or have tested against main/nightly)

Description

Vertex AI can return a valid outputInfo.gcsOutputDirectory longer than 255 characters for a completed batch prediction job. Bifrost maps that value to TableProviderJob.OutputFileID, whose PostgreSQL type is VARCHAR(255). The retrieved-job upsert then fails before batch accounting can settle.

In the observed case:

• Caller-supplied outputUriPrefix: 231 characters
• Vertex-appended output-directory suffix: 44 characters
• Returned outputInfo.gcsOutputDirectory: 275 characters
• PostgreSQL error: value too long for type character varying(255) (SQLSTATE 22001)

The Vertex batch itself completed successfully. However, Bifrost repeatedly failed to persist the retrieved job, so token/cost settlement and ReportBatchUsage never ran. Each affected batch is omitted from budget usage indefinitely, so subsequent requests may be admitted after actual provider spend exceeds the configured limit.

The same 255-character constraint is applied to all three opaque provider file identifiers:

• input_file_id
• output_file_id
• error_file_id

Relevant source:

• Vertex maps GcsOutputDirectory to OutputFileID: https://github.com/maximhq/bifrost/blob/e4a30d6041c0446603aea615bc5da340dac001b1/core/providers/vertex/batch.go
• TableProviderJob declares these identifiers as varchar(255): https://github.com/maximhq/bifrost/blob/e4a30d6041c0446603aea615bc5da340dac001b1/framework/configstore/tables/batchjob.go
• PostgreSQL migrations create the corresponding columns as VARCHAR(255): https://github.com/maximhq/bifrost/blob/e4a30d6041c0446603aea615bc5da340dac001b1/framework/configstore/migrations.go

Suggested fix: store opaque provider file identifiers as TEXT, add a forward PostgreSQL migration for existing databases, and cover identifiers longer than 255 characters in PostgreSQL batch-accounting tests. Truncating the identifiers would lose provider identity and should not be used as a fix.

This was reproduced on transports/v2.0.0. transports/v2.1.1 and the current dev branch were source-inspected and still declare the same varchar(255) fields, but were not runtime-tested.

Steps to reproduce

1. Configure Bifrost with PostgreSQL and a Vertex AI provider.
2. Submit a Vertex batch prediction job whose outputUriPrefix is long enough that Vertex's returned outputInfo.gcsOutputDirectory exceeds 255 characters. For example, a 231-character prefix plus a 44-character Vertex suffix produces a 275-character returned value.
3. Wait for the Vertex job to complete successfully.
4. Let Bifrost's delayed batch-accounting sweeper retrieve the completed job.
5. Observe that the retrieved-job upsert fails with SQLSTATE 22001 and repeats instead of settling the batch.
6. Observe that no batch usage/cost settlement is recorded and the affected spend is not applied to the virtual key's budget usage.

Expected behavior

Bifrost persists the complete provider-returned file identifiers, settles the completed batch, records token/cost usage, and applies the settled spend to the relevant virtual-key budget.

Actual behavior

Bifrost cannot persist a provider-returned identifier longer than 255 characters, repeatedly fails before settlement, and omits the affected batch's token/cost and virtual-key budget usage.

Affected area(s)

• Core (Go)
• Framework

Version

transports/v2.0.0 (reproduced); transports/v2.1.1 and current dev remain affected by source inspection.

Environment

- Database: PostgreSQL
- Provider: Vertex AI batch prediction
- Bifrost transport: v2.0.0

Relevant logs/output

value too long for type character varying(255) (SQLSTATE 22001)

Regression?

Unknown. The PostgreSQL schema currently uses VARCHAR(255) in v2.0.0, v2.1.1, and current dev.

Severity

High (major functionality broken)