Merge remote-tracking branch 'finos/feat/postgres' into feat/postgres-data-migration

# Conflicts:
#	website/docs/architecture/architecture.md

Author: dcoric

.agents/skills/sink-parity/SKILL.md

@@ -0,0 +1,69 @@
+---
+name: sink-parity
+description: Keep the fs, mongo and postgres sink backends at feature parity when changing any of them
+---
+
+Keep the sink backends at feature parity.
+
+Read `AGENTS.md` first. It is the canonical project guide for this repository.
+
+GitProxy persists its state through interchangeable sink backends: `src/db/file` (NeDB), `src/db/mongo`, and `src/db/postgres`. They all implement the `Sink` interface in `src/db/types.ts`, and deployments pick one via the `sink` config. A feature that exists in one backend but not the others is a bug waiting for whichever deployment uses the others.
+
+Use this skill whenever a change touches any of:
+
+- the `Sink` interface or the entity classes (`Repo`, `User`, push types) in `src/db/types.ts`
+- any backend adapter under `src/db/file`, `src/db/mongo`, or `src/db/postgres`
+- the migration framework (`src/db/migrations`) or the postgres schema
+
+## The contract
+
+- `src/db/types.ts` is the single source of truth: the `Sink` interface plus the doc comments on its members define the behaviour every backend must provide. No backend implementation, mongo included, outranks the contract. A new `Sink` member or entity field is not done until all three backends implement it in the same change; do not leave a backend behind for a follow-up.
+- `npm run check-types:server` enforces the interface structurally, but it cannot see semantic drift. The rest of this checklist exists for what the compiler cannot catch.
+
+## Adding or changing a Sink member
+
+1. Add the member to the `Sink` interface with a doc comment stating its semantics (ordering, case sensitivity, empty-result shape).
+2. Implement it in `src/db/file`, `src/db/mongo`, and `src/db/postgres`. The doc comment on the interface member is the reference for behaviour, not any one backend: whichever implementation lands first defines the semantics, so spell them out in the doc comment and make the other backends match it. Historically most members appeared in mongo first, but new functionality can just as well start in postgres; do not assume mongo is the template.
+3. Export it from each backend's `index.ts` and wire the dispatcher in `src/db/index.ts`.
+4. Add unit tests for every backend, not just the one you started from.
+
+## Adding a field to an entity
+
+1. Update the class in `src/db/types.ts`.
+2. fs and mongo store documents whole, so writes usually pass new fields through automatically; verify reads return them.
+3. postgres maps fields to columns explicitly, so every layer must be updated by hand:
+   - schema: add the column (see the migration rules below)
+   - create: insert the field, applying the same defaults as mongo (for example `dateCreated`/`lastModified` are stamped with the current ISO time on create)
+   - update: extend the column allowlist; a field missing from the allowlist is dropped silently, and an update reduced to zero columns throws, which has already nearly shipped a startup crash (`populateRepoDates`)
+   - read: add the column to every select and to the row-to-entity mapping
+4. If mongo or fs bump `lastModified` (or similar) on a mutation, every backend must bump it on that mutation.
+
+## Postgres schema changes
+
+- Schema changes are append-only migrations; never edit or reorder an entry that has shipped.
+- Cross-backend logical migrations belong in `src/db/migrations` (registered in `registry.ts`) and run through the `Sink` hooks (`getAppliedMigrations`, `recordMigration`, `unrecordMigration`), so they must work against all three backends.
+- `deriveCreatedAt` is best-effort by design: mongo derives a timestamp from the ObjectId, fs and postgres return `undefined` and callers fall back. Do not assume it returns a value.
+
+## Semantic parity rules
+
+- Same defaults on create in every backend.
+- Same case handling: usernames are lowercased on permission changes; name lookups are case-insensitive where mongo's are.
+- Same projections: list endpoints must return the same field set from every backend, or UI behaviour diverges by deployment.
+- Same error behaviour for invalid input (missing id, empty update).
+
+## Tests
+
+- Each backend has unit tests under `test/db/<backend>` that mock the driver (`pg` is mocked for postgres, NeDB runs in-memory for fs). Integration tests (`test/db/postgres/*.integration.test.ts`, `test/db/mongo/*.integration.test.ts`) run against a real service and are skipped when none is reachable; CI runs them in dedicated lanes.
+- When you touch a backend, add or extend BOTH kinds for it: unit coverage for the logic, integration coverage for the real query shapes. The fs backend has no external service, so unit coverage is enough there.
+- Keep the test scenarios aligned across backends: perform the same operations with the same inputs, and assert the same outputs, so parity is something the suite proves rather than something reviewers eyeball. If a scenario genuinely does not apply to a backend, say so in a comment instead of silently skipping it.
+
+## Verify before pushing
+
+```
+npm run check-types:server
+cross-env NODE_ENV=test npx vitest --run test/db
+npm run lint
+npm run format:check
+```
+
+Postgres integration tests (`npm run test:integration:postgres`) need a reachable PostgreSQL database; CI runs them in the dedicated lane.

.opencode/commands/sink-parity.md

@@ -0,0 +1,5 @@
+---
+description: Keep the fs, mongo and postgres sink backends at feature parity
+---
+
+Follow @.agents/skills/sink-parity/SKILL.md.

AGENTS.md

@@ -375,6 +375,10 @@ This file is the canonical project guide. Tool-specific entry points:
 - OpenCode: `.opencode/commands/`
 - Codex: `AGENTS.md`
 
+### Database changes
+
+The fs, mongo and postgres sink backends must stay at feature parity. Before changing anything under `src/db`, read `.agents/skills/sink-parity/SKILL.md`: apparently simple adaptor changes usually cost more on the postgres side (explicit columns, update allowlists, append-only schema migrations) than on the document stores, and the skill carries the checklist that keeps the backends aligned.
+
 ---
 
 ## Agent Workflow

src/db/postgres/helper.ts

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-import { Pool, PoolConfig, QueryResult, QueryResultRow } from 'pg';
+import { Pool, PoolClient, PoolConfig, QueryResult, QueryResultRow } from 'pg';
 import session, { Store } from 'express-session';
 import connectPgSimple from 'connect-pg-simple';
 
@@ -127,6 +127,31 @@ export const connect = async (): Promise<Pool> => {
   return pool;
 };
 
+/**
+ * Run `fn` inside a single transaction: every statement issued through the
+ * supplied client commits or rolls back together. Used where one logical
+ * update spans several statements, so a failure cannot leave partial state.
+ */
+export const withTransaction = async <T>(fn: (client: PoolClient) => Promise<T>): Promise<T> => {
+  const pool = await connect();
+  const client = await pool.connect();
+  try {
+    await client.query('BEGIN');
+    const result = await fn(client);
+    await client.query('COMMIT');
+    return result;
+  } catch (err) {
+    try {
+      await client.query('ROLLBACK');
+    } catch {
+      // the original error is the one worth surfacing
+    }
+    throw err;
+  } finally {
+    client.release();
+  }
+};
+
 export const query = async <T extends QueryResultRow = QueryResultRow>(
   text: string,
   params?: ReadonlyArray<unknown>,