diff --git a/.agents/skills/sink-parity/SKILL.md b/.agents/skills/sink-parity/SKILL.md new file mode 100644 index 000000000..28eb53cd0 --- /dev/null +++ b/.agents/skills/sink-parity/SKILL.md @@ -0,0 +1,69 @@ +--- +name: sink-parity +description: Keep the fs, mongo and postgres sink backends at feature parity when changing any of them +--- + +Keep the sink backends at feature parity. + +Read `AGENTS.md` first. It is the canonical project guide for this repository. + +GitProxy persists its state through interchangeable sink backends: `src/db/file` (NeDB), `src/db/mongo`, and `src/db/postgres`. They all implement the `Sink` interface in `src/db/types.ts`, and deployments pick one via the `sink` config. A feature that exists in one backend but not the others is a bug waiting for whichever deployment uses the others. + +Use this skill whenever a change touches any of: + +- the `Sink` interface or the entity classes (`Repo`, `User`, push types) in `src/db/types.ts` +- any backend adapter under `src/db/file`, `src/db/mongo`, or `src/db/postgres` +- the migration framework (`src/db/migrations`) or the postgres schema + +## The contract + +- `src/db/types.ts` is the single source of truth: the `Sink` interface plus the doc comments on its members define the behaviour every backend must provide. No backend implementation, mongo included, outranks the contract. A new `Sink` member or entity field is not done until all three backends implement it in the same change; do not leave a backend behind for a follow-up. +- `npm run check-types:server` enforces the interface structurally, but it cannot see semantic drift. The rest of this checklist exists for what the compiler cannot catch. + +## Adding or changing a Sink member + +1. Add the member to the `Sink` interface with a doc comment stating its semantics (ordering, case sensitivity, empty-result shape). +2. Implement it in `src/db/file`, `src/db/mongo`, and `src/db/postgres`. The doc comment on the interface member is the reference for behaviour, not any one backend: whichever implementation lands first defines the semantics, so spell them out in the doc comment and make the other backends match it. Historically most members appeared in mongo first, but new functionality can just as well start in postgres; do not assume mongo is the template. +3. Export it from each backend's `index.ts` and wire the dispatcher in `src/db/index.ts`. +4. Add unit tests for every backend, not just the one you started from. + +## Adding a field to an entity + +1. Update the class in `src/db/types.ts`. +2. fs and mongo store documents whole, so writes usually pass new fields through automatically; verify reads return them. +3. postgres maps fields to columns explicitly, so every layer must be updated by hand: + - schema: add the column (see the migration rules below) + - create: insert the field, applying the same defaults as mongo (for example `dateCreated`/`lastModified` are stamped with the current ISO time on create) + - update: extend the column allowlist; a field missing from the allowlist is dropped silently, and an update reduced to zero columns throws, which has already nearly shipped a startup crash (`populateRepoDates`) + - read: add the column to every select and to the row-to-entity mapping +4. If mongo or fs bump `lastModified` (or similar) on a mutation, every backend must bump it on that mutation. + +## Postgres schema changes + +- Schema changes are append-only migrations; never edit or reorder an entry that has shipped. +- Cross-backend logical migrations belong in `src/db/migrations` (registered in `registry.ts`) and run through the `Sink` hooks (`getAppliedMigrations`, `recordMigration`, `unrecordMigration`), so they must work against all three backends. +- `deriveCreatedAt` is best-effort by design: mongo derives a timestamp from the ObjectId, fs and postgres return `undefined` and callers fall back. Do not assume it returns a value. + +## Semantic parity rules + +- Same defaults on create in every backend. +- Same case handling: usernames are lowercased on permission changes; name lookups are case-insensitive where mongo's are. +- Same projections: list endpoints must return the same field set from every backend, or UI behaviour diverges by deployment. +- Same error behaviour for invalid input (missing id, empty update). + +## Tests + +- Each backend has unit tests under `test/db/` that mock the driver (`pg` is mocked for postgres, NeDB runs in-memory for fs). Integration tests (`test/db/postgres/*.integration.test.ts`, `test/db/mongo/*.integration.test.ts`) run against a real service and are skipped when none is reachable; CI runs them in dedicated lanes. +- When you touch a backend, add or extend BOTH kinds for it: unit coverage for the logic, integration coverage for the real query shapes. The fs backend has no external service, so unit coverage is enough there. +- Keep the test scenarios aligned across backends: perform the same operations with the same inputs, and assert the same outputs, so parity is something the suite proves rather than something reviewers eyeball. If a scenario genuinely does not apply to a backend, say so in a comment instead of silently skipping it. + +## Verify before pushing + +``` +npm run check-types:server +cross-env NODE_ENV=test npx vitest --run test/db +npm run lint +npm run format:check +``` + +Postgres integration tests (`npm run test:integration:postgres`) need a reachable PostgreSQL database; CI runs them in the dedicated lane. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d2d78065a..58f883ee2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -22,6 +22,24 @@ jobs: node-version: [22.x, 24.x] mongodb-version: ['6.0', '7.0', '8.0'] + # PostgreSQL service container for the postgres integration tests. A + # single version (postgres:16) keeps the lane fast; a broader version + # matrix can follow later if needed. + services: + postgres: + image: postgres:16 + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: git_proxy_test + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U postgres" + --health-interval 10s + --health-timeout 5s + --health-retries 5 + steps: - name: Harden Runner uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 @@ -71,6 +89,12 @@ jobs: GIT_PROXY_MONGO_CONNECTION_STRING: mongodb://localhost:27017/git-proxy-test run: npm run test:integration + - name: PostgreSQL Integration Tests + env: + RUN_POSTGRES_TESTS: 'true' + GIT_PROXY_POSTGRES_CONNECTION_STRING: postgresql://postgres:postgres@localhost:5432/git_proxy_test + run: npm run test:integration:postgres + - name: Upload test coverage report uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0 with: diff --git a/.opencode/commands/sink-parity.md b/.opencode/commands/sink-parity.md new file mode 100644 index 000000000..bdc240059 --- /dev/null +++ b/.opencode/commands/sink-parity.md @@ -0,0 +1,5 @@ +--- +description: Keep the fs, mongo and postgres sink backends at feature parity +--- + +Follow @.agents/skills/sink-parity/SKILL.md. diff --git a/AGENTS.md b/AGENTS.md index daa22656a..426e79baa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -375,6 +375,10 @@ This file is the canonical project guide. Tool-specific entry points: - OpenCode: `.opencode/commands/` - Codex: `AGENTS.md` +### Database changes + +The fs, mongo and postgres sink backends must stay at feature parity. Before changing anything under `src/db`, read `.agents/skills/sink-parity/SKILL.md`: apparently simple adaptor changes usually cost more on the postgres side (explicit columns, update allowlists, append-only schema migrations) than on the document stores, and the skill carries the checklist that keeps the backends aligned. + --- ## Agent Workflow diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6d09f1be5..9f0978866 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,6 +12,8 @@ For project governance, roles, and voting procedures, see the [Governance sectio - [Development Workflow](#development-workflow) - [Testing](#testing) - [Unit Tests](#unit-tests) + - [MongoDB Integration Tests](#mongodb-integration-tests) + - [PostgreSQL Integration Tests](#postgresql-integration-tests) - [End-to-End Tests](#end-to-end-tests) - [UI Tests (Cypress)](#ui-tests-cypress) - [Fuzz Tests](#fuzz-tests) @@ -87,7 +89,7 @@ git-proxy/ ├── src/ │ ├── proxy/ # Core proxy logic (action chain, processors) │ ├── service/ # Express app, API routes, authentication (Passport.js) -│ ├── db/ # Database abstraction (MongoDB + NeDB) +│ ├── db/ # Database abstraction (MongoDB, PostgreSQL, NeDB) │ ├── config/ # Configuration loading and generated types │ ├── ui/ # React dashboard (Material-UI) │ ├── plugin.ts # Plugin base classes (PushActionPlugin, PullActionPlugin) @@ -113,7 +115,7 @@ git-proxy/ - **Action chain**: Git push/fetch requests flow through a chain of processors in `src/proxy/chain.ts` - **Plugin system**: Extends the action chain with custom logic (see `src/plugin.ts`) -- **Dual database**: MongoDB for production state; [NeDB](https://github.com/seald/nedb) for local file-based development (`.data/` directory) +- **Pluggable database**: MongoDB or PostgreSQL for production state; [NeDB](https://github.com/seald/nedb) for local file-based development (`.data/` directory) - **Authentication**: Passport.js strategies (local, Active Directory, OpenID Connect) ## Development Workflow @@ -202,6 +204,31 @@ Configuration: [vitest.config.integration.ts](vitest.config.integration.ts) In CI, `RUN_MONGO_TESTS` is set automatically in the workflow that runs integration tests. +### PostgreSQL Integration Tests + +Some tests require a real PostgreSQL instance. These are guarded by the `RUN_POSTGRES_TESTS` environment variable and run separately from unit tests. + +```bash +# Start PostgreSQL with Docker +docker run -d --name postgres-test -p 5432:5432 \ + -e POSTGRES_USER=postgres \ + -e POSTGRES_PASSWORD=postgres \ + -e POSTGRES_DB=git_proxy_test \ + postgres:16 + +# Run PostgreSQL integration tests +npm run test:integration:postgres + +# Cleanup +docker stop postgres-test && docker rm postgres-test +``` + +Configuration: [vitest.config.integration.postgres.ts](vitest.config.integration.postgres.ts) + +Unlike the MongoDB lane, `RUN_POSTGRES_TESTS` and the connection string are set by the Vitest config itself, so no extra environment variables are required on the command line — you only need a PostgreSQL instance reachable at `postgresql://postgres:postgres@localhost:5432/git_proxy_test`. + +In CI, the PostgreSQL integration tests run against a `postgres:16` service container in the same workflow as the MongoDB integration tests. + ### End-to-End Tests E2E tests perform real git operations against a Dockerized environment. They use Vitest with a separate config. diff --git a/config.schema.json b/config.schema.json index a54d828d4..e4a6dba78 100644 --- a/config.schema.json +++ b/config.schema.json @@ -568,6 +568,81 @@ "enabled": { "type": "boolean" } }, "required": ["type", "enabled"] + }, + { + "type": "object", + "name": "PostgreSQL Config", + "description": "Connection properties for PostgreSQL. The `connectionString` may also be supplied via the `GIT_PROXY_POSTGRES_CONNECTION_STRING` environment variable. If neither a `connectionString` nor the discrete `host`/`port`/`user`/`password`/`database` fields are set, the standard `PGHOST`/`PGPORT`/`PGUSER`/`PGPASSWORD`/`PGDATABASE` environment variables are used.", + "properties": { + "type": { "type": "string", "const": "postgres" }, + "enabled": { "type": "boolean" }, + "connectionString": { + "type": "string", + "description": "PostgreSQL client connection string, see [https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING). If omitted, `GIT_PROXY_POSTGRES_CONNECTION_STRING` is used as a fallback, then the discrete fields below, then the `PG*` environment variables. Takes precedence over the discrete fields when set." + }, + "host": { + "type": "string", + "description": "Database server host. Used when `connectionString` is not set. Falls back to the `PGHOST` environment variable." + }, + "port": { + "type": "number", + "description": "Database server port. Used when `connectionString` is not set. Falls back to the `PGPORT` environment variable." + }, + "user": { + "type": "string", + "description": "Database user. Used when `connectionString` is not set. Falls back to the `PGUSER` environment variable." + }, + "password": { + "type": "string", + "description": "Database password. Used when `connectionString` is not set. Falls back to the `PGPASSWORD` environment variable." + }, + "database": { + "type": "string", + "description": "Database name. Used when `connectionString` is not set. Falls back to the `PGDATABASE` environment variable." + }, + "autoMigrate": { + "type": "boolean", + "description": "Run pending schema migrations automatically at startup (default `true`). Set to `false` when the runtime database role must not hold DDL rights: apply migrations out-of-band with DDL-capable credentials (`npm run migrate:postgres:schema`), and startup will only verify the schema is current, refusing to start while any migration is pending." + }, + "ssl": { + "description": "TLS configuration for the connection. `true` enables TLS with default certificate verification; an object is passed to the PostgreSQL client as TLS options (for example `rejectUnauthorized`, `ca`, `cert`, `key`).", + "oneOf": [{ "type": "boolean" }, { "type": "object", "additionalProperties": true }] + }, + "pool": { + "type": "object", + "description": "Connection pool tuning passed to the PostgreSQL client.", + "properties": { + "max": { + "type": "number", + "description": "Maximum number of clients the pool may hold." + }, + "idleTimeoutMillis": { + "type": "number", + "description": "Milliseconds a client may sit idle in the pool before being closed." + }, + "connectionTimeoutMillis": { + "type": "number", + "description": "Milliseconds to wait for a connection before timing out." + } + } + }, + "awsIamAuth": { + "type": "object", + "description": "Authenticate to Amazon RDS/Aurora with an IAM auth token instead of a static password. When enabled, a short-lived token is generated for each new connection from the AWS SDK default credential chain, so no password is stored. Requires the discrete `host`/`port`/`user` fields (or the `PGHOST`/`PGPORT`/`PGUSER` environment variables) rather than a `connectionString`, requires TLS (`ssl` defaults to `true` when omitted), and needs the optional `@aws-sdk/rds-signer` dependency to be installed.", + "properties": { + "enabled": { + "type": "boolean", + "description": "Enable IAM token authentication for the PostgreSQL connection." + }, + "region": { + "type": "string", + "description": "AWS region of the RDS/Aurora instance. Falls back to the `AWS_REGION` / `AWS_DEFAULT_REGION` environment variables, then the AWS SDK's default region resolution." + } + }, + "required": ["enabled"] + } + }, + "required": ["type", "enabled"] } ] }, diff --git a/eslint.config.mjs b/eslint.config.mjs index b074b622f..510ca046b 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -57,6 +57,8 @@ export default defineConfig( // vendored code we're not changing 'src/ui/assets/js/**', 'src/ui/assets/css/**', + // local claude worktrees / scratch + '.claude/**', ], }, diff --git a/package-lock.json b/package-lock.json index 7995b6bdd..5aa6db9b1 100644 --- a/package-lock.json +++ b/package-lock.json @@ -26,6 +26,7 @@ "clsx": "^2.1.1", "concurrently": "^9.2.4", "connect-mongo": "^6.0.0", + "connect-pg-simple": "^10.0.0", "cors": "^2.8.6", "diff2html": "^3.4.56", "env-paths": "^4.0.0", @@ -50,6 +51,7 @@ "passport": "^0.7.0", "passport-activedirectory": "^1.4.0", "passport-local": "^1.0.0", + "pg": "^8.20.0", "react": "^19.2.5", "react-dom": "^19.2.5", "react-is": "^19.0.0", @@ -74,6 +76,7 @@ "@eslint/json": "^2.0.0", "@tailwindcss/vite": "^4.2.2", "@types/activedirectory2": "^1.2.6", + "@types/connect-pg-simple": "^7.0.3", "@types/cors": "^2.8.19", "@types/express": "^5.0.6", "@types/express-http-proxy": "^1.6.7", @@ -85,6 +88,7 @@ "@types/node": "^22.19.7", "@types/passport": "^1.0.17", "@types/passport-local": "^1.0.38", + "@types/pg": "^8.20.0", "@types/react": "^19.2.14", "@types/react-dom": "^19.2.3", "@types/ssh2": "^1.15.5", @@ -121,6 +125,7 @@ "node": ">=22.13.1 || >=24.0.0" }, "optionalDependencies": { + "@aws-sdk/rds-signer": "^3.980.0", "@esbuild/darwin-arm64": "^0.27.2", "@esbuild/darwin-x64": "^0.27.2", "@esbuild/linux-x64": "0.27.2", @@ -356,6 +361,52 @@ "node": ">=20.0.0" } }, + "node_modules/@aws-sdk/rds-signer": { + "version": "3.1116.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/rds-signer/-/rds-signer-3.1116.0.tgz", + "integrity": "sha512-aaCORzUpgNb9oW0HkcY0fdSaXbbDFMhLUN7BkVyLZlxDBbbBi5PrZhUSd7gqOGYPdOmxeR0uNhsY6e5AvwMkkQ==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/credential-providers": "3.1116.0", + "@smithy/core": "^3.33.3", + "@smithy/signature-v4": "^5.6.12", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/rds-signer/node_modules/@aws-sdk/credential-providers": { + "version": "3.1116.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-providers/-/credential-providers-3.1116.0.tgz", + "integrity": "sha512-y41rRJ1AWtcJka2YdFQ1BfTf0CZDXizh0VOZLwfUzxI2xhG7n88XXn0N0yvLwI73/tjtXalVy94/N5QCO1lgbg==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/credential-provider-cognito-identity": "^3.972.69", + "@aws-sdk/credential-provider-env": "^3.972.70", + "@aws-sdk/credential-provider-http": "^3.972.72", + "@aws-sdk/credential-provider-ini": "^3.973.15", + "@aws-sdk/credential-provider-login": "^3.972.77", + "@aws-sdk/credential-provider-node": "^3.972.81", + "@aws-sdk/credential-provider-process": "^3.972.70", + "@aws-sdk/credential-provider-sso": "^3.973.14", + "@aws-sdk/credential-provider-web-identity": "^3.972.76", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/credential-provider-imds": "^4.4.16", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, "node_modules/@aws-sdk/signature-v4-multi-region": { "version": "3.996.46", "resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.46.tgz", @@ -2217,18 +2268,6 @@ "react-dom": "^16.8.0 || ^17.0.0" } }, - "node_modules/@finos/git-proxy/node_modules/@types/react": { - "version": "17.0.93", - "resolved": "https://registry.npmjs.org/@types/react/-/react-17.0.93.tgz", - "integrity": "sha512-KM4Ty/ZTLZupiYxZVAlP+InNJS3De6uBMdq0ePa6/04+eG9Y7ftnWfst1xTLQ5rwAhgHwQ4momt/O4KepdGBTw==", - "extraneous": true, - "license": "MIT", - "dependencies": { - "@types/prop-types": "*", - "@types/scheduler": "^0.16", - "csstype": "^3.2.2" - } - }, "node_modules/@finos/git-proxy/node_modules/dom-serializer": { "version": "0.2.2", "resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-0.2.2.tgz", @@ -3470,9 +3509,6 @@ "arm64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -3490,9 +3526,6 @@ "arm64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -3510,9 +3543,6 @@ "ppc64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -3530,9 +3560,6 @@ "s390x" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -3550,9 +3577,6 @@ "x64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -3570,9 +3594,6 @@ "x64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -3920,9 +3941,6 @@ "arm64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -3940,9 +3958,6 @@ "arm64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -3960,9 +3975,6 @@ "x64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -3980,9 +3992,6 @@ "x64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -4022,6 +4031,72 @@ "node": ">=14.0.0" } }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/core": { + "version": "1.11.1", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.2.2", + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/runtime": { + "version": "1.11.1", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/wasi-threads": { + "version": "1.2.2", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@napi-rs/wasm-runtime": { + "version": "1.1.4", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@tybys/wasm-util": "^0.10.1" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "peerDependencies": { + "@emnapi/core": "^1.7.1", + "@emnapi/runtime": "^1.7.1" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@tybys/wasm-util": { + "version": "0.10.2", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/tslib": { + "version": "2.8.1", + "dev": true, + "inBundle": true, + "license": "0BSD", + "optional": true + }, "node_modules/@tailwindcss/oxide-win32-arm64-msvc": { "version": "4.3.3", "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-win32-arm64-msvc/-/oxide-win32-arm64-msvc-4.3.3.tgz", @@ -4205,6 +4280,18 @@ "@types/node": "*" } }, + "node_modules/@types/connect-pg-simple": { + "version": "7.0.3", + "resolved": "https://registry.npmjs.org/@types/connect-pg-simple/-/connect-pg-simple-7.0.3.tgz", + "integrity": "sha512-NGCy9WBlW2bw+J/QlLnFZ9WjoGs6tMo3LAut6mY4kK+XHzue//lpNVpAvYRpIwM969vBRAM2Re0izUvV6kt+NA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/express": "*", + "@types/express-session": "*", + "@types/pg": "*" + } + }, "node_modules/@types/conventional-commits-parser": { "version": "5.0.2", "resolved": "https://registry.npmjs.org/@types/conventional-commits-parser/-/conventional-commits-parser-5.0.2.tgz", @@ -4413,12 +4500,17 @@ "@types/passport": "*" } }, - "node_modules/@types/prop-types": { - "version": "15.7.15", - "resolved": "https://registry.npmjs.org/@types/prop-types/-/prop-types-15.7.15.tgz", - "integrity": "sha512-F6bEyamV9jKGAFBEmlQnesRPGOQqS2+Uwi0Em15xenOxHaf2hv6L8YCVn3rPdPJOiJfPiCnLIRyvwVaqMY3MIw==", - "extraneous": true, - "license": "MIT" + "node_modules/@types/pg": { + "version": "8.23.1", + "resolved": "https://registry.npmjs.org/@types/pg/-/pg-8.23.1.tgz", + "integrity": "sha512-fKVHpikPdg4GKks3JuLEhvwSyvwzF23hnabPy6DD8ljVbC7+6J5dQzdv4arV6jqq57djnMgs1HKBxX4P8aBI3A==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/node": "*", + "pg-protocol": "*", + "pg-types": "^2.2.0" + } }, "node_modules/@types/qs": { "version": "6.15.1", @@ -4462,13 +4554,6 @@ "@types/react": "*" } }, - "node_modules/@types/scheduler": { - "version": "0.16.8", - "resolved": "https://registry.npmjs.org/@types/scheduler/-/scheduler-0.16.8.tgz", - "integrity": "sha512-WZLiwShhwLRmeV6zH+GkbOFT6Z6VklCItrDioxUnv+u4Ll+8vKeFySoFyK/0ctcRpOmwAicELfmys1sDc/Rw+A==", - "extraneous": true, - "license": "MIT" - }, "node_modules/@types/send": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/@types/send/-/send-1.2.1.tgz", @@ -6730,6 +6815,18 @@ "mongodb": ">=5.0.0" } }, + "node_modules/connect-pg-simple": { + "version": "10.0.0", + "resolved": "https://registry.npmjs.org/connect-pg-simple/-/connect-pg-simple-10.0.0.tgz", + "integrity": "sha512-pBGVazlqiMrackzCr0eKhn4LO5trJXsOX0nQoey9wCOayh80MYtThCbq8eoLsjpiWgiok/h+1/uti9/2/Una8A==", + "license": "MIT", + "dependencies": { + "pg": "^8.12.0" + }, + "engines": { + "node": "^18.18.0 || ^20.9.0 || >=22.0.0" + } + }, "node_modules/content-disposition": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz", @@ -11181,9 +11278,6 @@ "arm64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -11205,9 +11299,6 @@ "arm64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -11229,9 +11320,6 @@ "x64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -11253,9 +11341,6 @@ "x64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -12765,6 +12850,95 @@ "dev": true, "license": "MIT" }, + "node_modules/pg": { + "version": "8.23.0", + "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz", + "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==", + "license": "MIT", + "dependencies": { + "pg-connection-string": "^2.14.0", + "pg-pool": "^3.14.0", + "pg-protocol": "^1.16.0", + "pg-types": "2.2.0", + "pgpass": "1.0.5" + }, + "engines": { + "node": ">= 16.0.0" + }, + "optionalDependencies": { + "pg-cloudflare": "^1.4.0" + }, + "peerDependencies": { + "pg-native": ">=3.0.1" + }, + "peerDependenciesMeta": { + "pg-native": { + "optional": true + } + } + }, + "node_modules/pg-cloudflare": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz", + "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==", + "license": "MIT", + "optional": true + }, + "node_modules/pg-connection-string": { + "version": "2.14.0", + "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz", + "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==", + "license": "MIT" + }, + "node_modules/pg-int8": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz", + "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==", + "license": "ISC", + "engines": { + "node": ">=4.0.0" + } + }, + "node_modules/pg-pool": { + "version": "3.14.0", + "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz", + "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==", + "license": "MIT", + "peerDependencies": { + "pg": ">=8.0" + } + }, + "node_modules/pg-protocol": { + "version": "1.16.0", + "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz", + "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==", + "license": "MIT" + }, + "node_modules/pg-types": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz", + "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==", + "license": "MIT", + "dependencies": { + "pg-int8": "1.0.1", + "postgres-array": "~2.0.0", + "postgres-bytea": "~1.0.0", + "postgres-date": "~1.0.4", + "postgres-interval": "^1.1.0" + }, + "engines": { + "node": ">=4" + } + }, + "node_modules/pgpass": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz", + "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==", + "license": "MIT", + "dependencies": { + "split2": "^4.1.0" + } + }, "node_modules/picocolors": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", @@ -12849,6 +13023,45 @@ "node": "^10 || ^12 || >=14" } }, + "node_modules/postgres-array": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz", + "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==", + "license": "MIT", + "engines": { + "node": ">=4" + } + }, + "node_modules/postgres-bytea": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz", + "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/postgres-date": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz", + "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/postgres-interval": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz", + "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==", + "license": "MIT", + "dependencies": { + "xtend": "^4.0.0" + }, + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/precond": { "version": "0.2.3", "resolved": "https://registry.npmjs.org/precond/-/precond-0.2.3.tgz", @@ -14272,7 +14485,6 @@ "version": "4.2.0", "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz", "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==", - "dev": true, "license": "ISC", "engines": { "node": ">= 10.x" @@ -15845,8 +16057,10 @@ "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", - "extraneous": true, + "dev": true, "license": "Apache-2.0", + "optional": true, + "peer": true, "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" @@ -15998,9 +16212,6 @@ "arm64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -16022,9 +16233,6 @@ "arm64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -16046,9 +16254,6 @@ "x64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -16070,9 +16275,6 @@ "x64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -16515,6 +16717,15 @@ } } }, + "node_modules/xtend": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz", + "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==", + "license": "MIT", + "engines": { + "node": ">=0.4" + } + }, "node_modules/y18n": { "version": "5.0.8", "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz", diff --git a/package.json b/package.json index d6aff453b..29161d7b2 100644 --- a/package.json +++ b/package.json @@ -54,6 +54,8 @@ "scripts": { "cli": "tsx ./packages/git-proxy-cli/index.ts", "cli:js": "node ./packages/git-proxy-cli/dist/index.js", + "migrate:postgres": "tsx scripts/migrate-to-postgres.ts", + "migrate:postgres:schema": "tsx scripts/migrate-postgres-schema.ts", "client": "vite --config vite.config.ts", "clientinstall": "npm install --prefix client", "server": "cross-env ALLOWED_ORIGINS=* tsx index.ts", @@ -70,6 +72,7 @@ "test-coverage": "cross-env NODE_ENV=test vitest --run --dir ./test --coverage", "test-coverage-ci": "cross-env NODE_ENV=test vitest --run --dir ./test --coverage.enabled=true --coverage.reporter=lcovonly --coverage.reporter=text", "test:integration": "cross-env NODE_ENV=test vitest --run --config vitest.config.integration.ts", + "test:integration:postgres": "cross-env NODE_ENV=test vitest --run --config vitest.config.integration.postgres.ts", "test:watch": "cross-env NODE_ENV=test vitest --dir ./test --watch", "test:migrate": "cross-env NODE_ENV=test vitest --run --dir ./scripts/migrate/test", "prepare": "node ./scripts/prepare.js", @@ -120,6 +123,7 @@ "clsx": "^2.1.1", "concurrently": "^9.2.4", "connect-mongo": "^6.0.0", + "connect-pg-simple": "^10.0.0", "cors": "^2.8.6", "diff2html": "^3.4.56", "env-paths": "^4.0.0", @@ -144,6 +148,7 @@ "passport": "^0.7.0", "passport-activedirectory": "^1.4.0", "passport-local": "^1.0.0", + "pg": "^8.20.0", "react": "^19.2.5", "react-dom": "^19.2.5", "react-is": "^19.0.0", @@ -164,6 +169,7 @@ "@eslint/json": "^2.0.0", "@tailwindcss/vite": "^4.2.2", "@types/activedirectory2": "^1.2.6", + "@types/connect-pg-simple": "^7.0.3", "@types/cors": "^2.8.19", "@types/express": "^5.0.6", "@types/express-http-proxy": "^1.6.7", @@ -175,6 +181,7 @@ "@types/node": "^22.19.7", "@types/passport": "^1.0.17", "@types/passport-local": "^1.0.38", + "@types/pg": "^8.20.0", "@types/react": "^19.2.14", "@types/react-dom": "^19.2.3", "@types/ssh2": "^1.15.5", @@ -212,6 +219,7 @@ "underscore": "^1.13.8" }, "optionalDependencies": { + "@aws-sdk/rds-signer": "^3.980.0", "@esbuild/darwin-arm64": "^0.27.2", "@esbuild/darwin-x64": "^0.27.2", "@esbuild/linux-x64": "0.27.2", diff --git a/proxy.config.json b/proxy.config.json index 5848348c6..2e2f5793c 100644 --- a/proxy.config.json +++ b/proxy.config.json @@ -39,6 +39,11 @@ "ssl": true }, "enabled": false + }, + { + "type": "postgres", + "connectionString": "postgresql://localhost:5432/gitproxy", + "enabled": false } ], "authentication": [ diff --git a/scripts/migrate-postgres-schema.ts b/scripts/migrate-postgres-schema.ts new file mode 100644 index 000000000..f2b9215df --- /dev/null +++ b/scripts/migrate-postgres-schema.ts @@ -0,0 +1,45 @@ +#!/usr/bin/env tsx + +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * Apply pending postgres schema migrations and exit. + * + * For deployments running with `autoMigrate: false`, where the runtime + * database role holds no DDL rights: run this with DDL-capable credentials + * (via the configured sink, or the standard PG* / connection-string + * environment overrides) before starting the new GitProxy version. + */ + +import { getDatabase } from '../src/config'; +import { applySchemaMigrations, resetConnection } from '../src/db/postgres/helper'; + +const main = async (): Promise => { + const db = getDatabase(); + if (db.type !== 'postgres') { + throw new Error(`the active sink is '${db.type}', not postgres — nothing to migrate`); + } + await applySchemaMigrations(); + console.log('postgres schema is up to date'); +}; + +main() + .catch((err) => { + console.error(err instanceof Error ? err.message : err); + process.exitCode = 1; + }) + .finally(() => resetConnection()); diff --git a/scripts/migrate-to-postgres.ts b/scripts/migrate-to-postgres.ts new file mode 100644 index 000000000..c54140da3 --- /dev/null +++ b/scripts/migrate-to-postgres.ts @@ -0,0 +1,87 @@ +#!/usr/bin/env tsx + +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import yargs from 'yargs'; +import { hideBin } from 'yargs/helpers'; + +import { getDatabase } from '../src/config'; +import * as postgres from '../src/db/postgres'; +import { resetConnection } from '../src/db/postgres/helper'; +import { migrate, MigrationSource } from '../src/db/postgres/migrate'; +import { createFileSource } from '../src/db/postgres/migrateFileSource'; +import { createMongoSource } from '../src/db/postgres/migrateMongoSource'; + +const argv = yargs(hideBin(process.argv)) + .usage('Usage: $0 --from [options]') + .option('from', { + choices: ['mongo', 'fs'] as const, + demandOption: true, + describe: 'Source backend to migrate from', + }) + .option('mongoUrl', { + type: 'string', + describe: 'MongoDB connection string (required when --from mongo)', + }) + .option('dataDir', { + type: 'string', + describe: 'NeDB data directory (defaults to ./.data/db) when --from fs', + }) + .strict() + .parseSync(); + +const buildSource = async (): Promise => { + if (argv.from === 'mongo') { + if (!argv.mongoUrl) { + throw new Error('--mongoUrl is required when --from mongo'); + } + return createMongoSource(argv.mongoUrl); + } + return createFileSource(argv.dataDir); +}; + +const main = async (): Promise => { + // The destination is the active sink, so it must be postgres. Reading the + // source is independent (its own driver), so the two never clash. + const db = getDatabase(); + if (db.type !== 'postgres') { + throw new Error( + `The active sink is "${db.type}", but this migration writes to postgres. ` + + 'Enable the postgres sink (with its connectionString or ' + + 'GIT_PROXY_POSTGRES_CONNECTION_STRING) before running this.', + ); + } + + const source = await buildSource(); + try { + const summary = await migrate(source, postgres, { log: (message) => console.log(message) }); + console.log('Migration complete:'); + console.log(` users: ${summary.users.imported} imported, ${summary.users.skipped} skipped`); + console.log(` repos: ${summary.repos.imported} imported, ${summary.repos.skipped} skipped`); + console.log(` pushes: ${summary.pushes.imported} imported`); + } finally { + await source.close(); + // Close the destination pool too, or its open handles keep the process + // alive after the summary prints. + await resetConnection(); + } +}; + +main().catch((err) => { + console.error(`Migration failed: ${err instanceof Error ? err.message : err}`); + process.exit(1); +}); diff --git a/src/config/env.ts b/src/config/env.ts index 503764ee1..aab264341 100644 --- a/src/config/env.ts +++ b/src/config/env.ts @@ -29,6 +29,7 @@ const { GIT_PROXY_HTTPS_UI_PORT, GIT_PROXY_COOKIE_SECRET, GIT_PROXY_MONGO_CONNECTION_STRING = 'mongodb://localhost:27017/git-proxy', + GIT_PROXY_POSTGRES_CONNECTION_STRING, } = process.env; export const serverConfig: ServerConfig = { @@ -39,4 +40,5 @@ export const serverConfig: ServerConfig = { GIT_PROXY_HTTPS_UI_PORT, GIT_PROXY_COOKIE_SECRET, GIT_PROXY_MONGO_CONNECTION_STRING, + GIT_PROXY_POSTGRES_CONNECTION_STRING, }; diff --git a/src/config/generated/config.ts b/src/config/generated/config.ts index 0bc0baf6a..90de3e0f1 100644 --- a/src/config/generated/config.ts +++ b/src/config/generated/config.ts @@ -538,11 +538,23 @@ export interface RateLimit { * or broken out in the options object * * Connection properties for an neDB file-based database + * + * Connection properties for PostgreSQL. The `connectionString` may also be supplied via the + * `GIT_PROXY_POSTGRES_CONNECTION_STRING` environment variable. If neither a + * `connectionString` nor the discrete `host`/`port`/`user`/`password`/`database` fields are + * set, the standard `PGHOST`/`PGPORT`/`PGUSER`/`PGPASSWORD`/`PGDATABASE` environment + * variables are used. */ export interface Database { /** * mongoDB Client connection string, see * [https://www.mongodb.com/docs/manual/reference/connection-string/](https://www.mongodb.com/docs/manual/reference/connection-string/) + * + * PostgreSQL client connection string, see + * [https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING). + * If omitted, `GIT_PROXY_POSTGRES_CONNECTION_STRING` is used as a fallback, then the + * discrete fields below, then the `PG*` environment variables. Takes precedence over the + * discrete fields when set. */ connectionString?: string; enabled: boolean; @@ -554,6 +566,78 @@ export interface Database { */ options?: Options; type: DatabaseType; + /** + * Run pending schema migrations automatically at startup (default `true`). Set to `false` + * when the runtime database role must not hold DDL rights: apply migrations out-of-band + * with DDL-capable credentials (`npm run migrate:postgres:schema`), and startup will only + * verify the schema is current, refusing to start while any migration is pending. + */ + autoMigrate?: boolean; + /** + * Authenticate to Amazon RDS/Aurora with an IAM auth token instead of a static password. + * When enabled, a short-lived token is generated for each new connection from the AWS SDK + * default credential chain, so no password is stored. Requires the discrete + * `host`/`port`/`user` fields (or the `PGHOST`/`PGPORT`/`PGUSER` environment variables) + * rather than a `connectionString`, requires TLS (`ssl` defaults to `true` when omitted), + * and needs the optional `@aws-sdk/rds-signer` dependency to be installed. + */ + awsIamAuth?: AwsIamAuth; + /** + * Database name. Used when `connectionString` is not set. Falls back to the `PGDATABASE` + * environment variable. + */ + database?: string; + /** + * Database server host. Used when `connectionString` is not set. Falls back to the `PGHOST` + * environment variable. + */ + host?: string; + /** + * Database password. Used when `connectionString` is not set. Falls back to the + * `PGPASSWORD` environment variable. + */ + password?: string; + /** + * Connection pool tuning passed to the PostgreSQL client. + */ + pool?: Pool; + /** + * Database server port. Used when `connectionString` is not set. Falls back to the `PGPORT` + * environment variable. + */ + port?: number; + /** + * TLS configuration for the connection. `true` enables TLS with default certificate + * verification; an object is passed to the PostgreSQL client as TLS options (for example + * `rejectUnauthorized`, `ca`, `cert`, `key`). + */ + ssl?: boolean | { [key: string]: any }; + /** + * Database user. Used when `connectionString` is not set. Falls back to the `PGUSER` + * environment variable. + */ + user?: string; + [property: string]: any; +} + +/** + * Authenticate to Amazon RDS/Aurora with an IAM auth token instead of a static password. + * When enabled, a short-lived token is generated for each new connection from the AWS SDK + * default credential chain, so no password is stored. Requires the discrete + * `host`/`port`/`user` fields (or the `PGHOST`/`PGPORT`/`PGUSER` environment variables) + * rather than a `connectionString`, requires TLS (`ssl` defaults to `true` when omitted), + * and needs the optional `@aws-sdk/rds-signer` dependency to be installed. + */ +export interface AwsIamAuth { + /** + * Enable IAM token authentication for the PostgreSQL connection. + */ + enabled: boolean; + /** + * AWS region of the RDS/Aurora instance. Falls back to the `AWS_REGION` / + * `AWS_DEFAULT_REGION` environment variables, then the AWS SDK's default region resolution. + */ + region?: string; [property: string]: any; } @@ -577,9 +661,29 @@ export interface AuthMechanismProperties { [property: string]: any; } +/** + * Connection pool tuning passed to the PostgreSQL client. + */ +export interface Pool { + /** + * Milliseconds to wait for a connection before timing out. + */ + connectionTimeoutMillis?: number; + /** + * Milliseconds a client may sit idle in the pool before being closed. + */ + idleTimeoutMillis?: number; + /** + * Maximum number of clients the pool may hold. + */ + max?: number; + [property: string]: any; +} + export enum DatabaseType { FS = 'fs', Mongo = 'mongo', + Postgres = 'postgres', } /** @@ -1111,6 +1215,22 @@ const typeMap: any = { { json: 'enabled', js: 'enabled', typ: true }, { json: 'options', js: 'options', typ: u(undefined, r('Options')) }, { json: 'type', js: 'type', typ: r('DatabaseType') }, + { json: 'autoMigrate', js: 'autoMigrate', typ: u(undefined, true) }, + { json: 'awsIamAuth', js: 'awsIamAuth', typ: u(undefined, r('AwsIamAuth')) }, + { json: 'database', js: 'database', typ: u(undefined, '') }, + { json: 'host', js: 'host', typ: u(undefined, '') }, + { json: 'password', js: 'password', typ: u(undefined, '') }, + { json: 'pool', js: 'pool', typ: u(undefined, r('Pool')) }, + { json: 'port', js: 'port', typ: u(undefined, 3.14) }, + { json: 'ssl', js: 'ssl', typ: u(undefined, u(true, m('any'))) }, + { json: 'user', js: 'user', typ: u(undefined, '') }, + ], + 'any', + ), + AwsIamAuth: o( + [ + { json: 'enabled', js: 'enabled', typ: true }, + { json: 'region', js: 'region', typ: u(undefined, '') }, ], 'any', ), @@ -1128,6 +1248,14 @@ const typeMap: any = { [{ json: 'AWS_CREDENTIAL_PROVIDER', js: 'AWS_CREDENTIAL_PROVIDER', typ: u(undefined, true) }], 'any', ), + Pool: o( + [ + { json: 'connectionTimeoutMillis', js: 'connectionTimeoutMillis', typ: u(undefined, 3.14) }, + { json: 'idleTimeoutMillis', js: 'idleTimeoutMillis', typ: u(undefined, 3.14) }, + { json: 'max', js: 'max', typ: u(undefined, 3.14) }, + ], + 'any', + ), SSH: o( [ { @@ -1200,6 +1328,6 @@ const typeMap: any = { false, ), AuthenticationElementType: ['ActiveDirectory', 'jwt', 'local', 'openidconnect'], - DatabaseType: ['fs', 'mongo'], + DatabaseType: ['fs', 'mongo', 'postgres'], AuthType: ['basic', 'ntlm'], }; diff --git a/src/config/index.ts b/src/config/index.ts index 6f6ac90b4..16f5ef313 100644 --- a/src/config/index.ts +++ b/src/config/index.ts @@ -329,6 +329,10 @@ export const getDatabase = () => { if (db.type === 'mongo' && !db.connectionString) { db.connectionString = serverConfig.GIT_PROXY_MONGO_CONNECTION_STRING; } + // same fallback for postgres + if (db.type === 'postgres' && !db.connectionString) { + db.connectionString = serverConfig.GIT_PROXY_POSTGRES_CONNECTION_STRING; + } return db; } } diff --git a/src/config/types.ts b/src/config/types.ts index 5cf5fd60c..d2065f4d5 100644 --- a/src/config/types.ts +++ b/src/config/types.ts @@ -24,6 +24,7 @@ export type ServerConfig = { GIT_PROXY_HTTPS_UI_PORT: string | undefined; GIT_PROXY_COOKIE_SECRET: string | undefined; GIT_PROXY_MONGO_CONNECTION_STRING: string; + GIT_PROXY_POSTGRES_CONNECTION_STRING: string | undefined; }; interface GitAuth { diff --git a/src/db/index.ts b/src/db/index.ts index bfd2f5ef7..10fb8dbf5 100644 --- a/src/db/index.ts +++ b/src/db/index.ts @@ -30,8 +30,10 @@ import * as bcrypt from 'bcryptjs'; import * as config from '../config'; import * as mongo from './mongo'; import * as neDb from './file'; +import * as postgres from './postgres'; import { Action } from '../proxy/actions/Action'; import MongoDBStore from 'connect-mongo'; +import { Store } from 'express-session'; import { CompletedAttestation, Rejection } from '../proxy/processors/types'; import { processGitUrl } from '../proxy/routes/helper'; import { initializeFolders } from './file/helper'; @@ -56,6 +58,9 @@ const start = () => { console.log('Loading neDB database adaptor'); initializeFolders(); _sink = neDb; + } else if (config.getDatabase().type === 'postgres') { + console.log('Loading PostgreSQL database adaptor'); + _sink = postgres; } else { console.error(`Unsupported database type: ${config.getDatabase().type}`); process.exit(1); @@ -197,7 +202,9 @@ export const canUserCancelPush = async (id: string, user: string) => { }; export const runMigrations = (): Promise => applyMigrations(start(), migrations); -export const getSessionStore = (): MongoDBStore | undefined => start().getSessionStore(); +export const getSessionStore = (): MongoDBStore | Store | undefined => start().getSessionStore(); +export const ensureSessionStoreReady = (): Promise => + start().ensureSessionStoreReady?.() ?? Promise.resolve(); export const getPushes = (query: Partial): Promise => start().getPushes(query); export const getPushesForUserProfile = async (user: User): Promise => { const emailVariants = collectUserProfileEmailVariants(user); diff --git a/src/db/postgres/helper.ts b/src/db/postgres/helper.ts new file mode 100644 index 000000000..4427c7521 --- /dev/null +++ b/src/db/postgres/helper.ts @@ -0,0 +1,343 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Pool, PoolClient, PoolConfig, QueryResult, QueryResultRow } from 'pg'; +import session, { Store } from 'express-session'; +import connectPgSimple from 'connect-pg-simple'; + +import { getDatabase } from '../../config'; +import { assertMigrationsCurrent, runMigrations } from './schemaMigrations'; + +type DatabaseConfig = ReturnType; + +let _pool: Pool | null = null; +let _bootstrapPromise: Promise | null = null; + +/** + * True when some Postgres connection is configured: an explicit connection + * string, the discrete `host` field, or any of the standard `PG*` environment + * variables that identify a target (`PGHOST`, `PGHOSTADDR`, `PGUSER`, + * `PGDATABASE`). Used to refuse startup loudly rather than silently defaulting + * to `localhost`. + */ +const hasConnectionConfig = (db: DatabaseConfig): boolean => + Boolean( + db.connectionString || + db.host || + process.env.PGHOST || + process.env.PGHOSTADDR || + process.env.PGUSER || + process.env.PGDATABASE, + ); + +/** + * Minimal shape of the optional `@aws-sdk/rds-signer` module, declared locally + * so the project type-checks whether or not the optional dependency is present. + */ +interface RdsSignerModule { + Signer: new (config: { hostname: string; port: number; username: string; region?: string }) => { + getAuthToken: () => Promise; + }; +} + +/** + * Load the optional RDS signer, raising a clear error if it is not installed. + * Kept optional so installs that do not use IAM auth stay lean. + */ +const loadRdsSigner = async (): Promise => { + try { + return (await import('@aws-sdk/rds-signer')) as unknown as RdsSignerModule; + } catch { + throw new Error( + 'AWS RDS IAM authentication requires the optional `@aws-sdk/rds-signer` dependency. Install it with `npm install @aws-sdk/rds-signer`.', + ); + } +}; + +/** + * Build the per-connection password provider for RDS/Aurora IAM auth. `pg` + * invokes it for every new connection, so each one receives a fresh (~15 min) + * token and refresh is automatic — no static password is ever stored. + */ +const buildIamTokenProvider = (db: DatabaseConfig): (() => Promise) => { + const host = db.host ?? process.env.PGHOST; + const port = db.port ?? (process.env.PGPORT ? Number(process.env.PGPORT) : 5432); + const user = db.user ?? process.env.PGUSER; + const region = db.awsIamAuth?.region ?? process.env.AWS_REGION ?? process.env.AWS_DEFAULT_REGION; + + if (!host || !user) { + throw new Error( + 'AWS RDS IAM authentication requires `host` and `user` (or the PGHOST/PGUSER environment variables) to generate an auth token.', + ); + } + + return async () => { + const { Signer } = await loadRdsSigner(); + const signer = new Signer({ hostname: host, port, username: user, region }); + return signer.getAuthToken(); + }; +}; + +const hasDiscreteFields = (db: DatabaseConfig): boolean => + db.host !== undefined || + db.port !== undefined || + db.user !== undefined || + db.password !== undefined || + db.database !== undefined; + +/** + * Copy whichever discrete connection fields are set onto the pool config. + * `password` is excluded in IAM mode, where a token provider replaces it. + */ +const applyDiscreteFields = ( + db: DatabaseConfig, + config: PoolConfig, + { includePassword }: { includePassword: boolean }, +): void => { + if (db.host !== undefined) config.host = db.host; + if (db.port !== undefined) config.port = db.port; + if (db.user !== undefined) config.user = db.user; + if (includePassword && db.password !== undefined) config.password = db.password; + if (db.database !== undefined) config.database = db.database; +}; + +// TLS applies regardless of how the connection itself was configured. RDS IAM +// auth mandates TLS, so default it on when IAM is enabled and `ssl` is unset. +const applySsl = (db: DatabaseConfig, config: PoolConfig, iamAuthEnabled: boolean): void => { + if (db.ssl !== undefined) { + config.ssl = db.ssl as PoolConfig['ssl']; + if (iamAuthEnabled && typeof db.ssl === 'object' && db.ssl !== null && !('ca' in db.ssl)) { + console.warn( + '[postgres] awsIamAuth: the ssl options carry no `ca`; RDS server certificates chain to ' + + "Amazon's RDS root CA, which is not in Node's default trust store, so verification " + + 'will fail unless the RDS CA bundle is supplied via ssl.ca', + ); + } + } else if (iamAuthEnabled) { + // RDS requires TLS for IAM auth, so it defaults on. `ssl: true` verifies + // against Node's default trust store, which does NOT contain Amazon's RDS + // root CA; connecting to a real RDS endpoint therefore needs the RDS CA + // bundle supplied via `ssl.ca`. The default stays verify-on rather than + // silently downgrading transport security. + config.ssl = true; + console.warn( + '[postgres] awsIamAuth: ssl defaulted to true, which verifies against ' + + "Node's default trust store; connections to RDS will fail certificate " + + 'verification unless the RDS CA bundle is supplied via ssl.ca ' + + '(see the PostgreSQL section of the architecture doc)', + ); + } +}; + +const applyPoolTuning = (db: DatabaseConfig, config: PoolConfig): void => { + if (!db.pool) return; + if (db.pool.max !== undefined) config.max = db.pool.max; + if (db.pool.idleTimeoutMillis !== undefined) { + config.idleTimeoutMillis = db.pool.idleTimeoutMillis; + } + if (db.pool.connectionTimeoutMillis !== undefined) { + config.connectionTimeoutMillis = db.pool.connectionTimeoutMillis; + } +}; + +/** + * Build a `pg` PoolConfig from the resolved database config. A connection + * string (already env-resolved by `getDatabase`) takes precedence; otherwise + * the discrete fields are used. When neither is set, `pg` reads the `PG*` + * environment variables itself. When `awsIamAuth` is enabled, the static + * password is replaced by a generated IAM token and the discrete fields drive + * the connection. + */ +const buildPoolConfig = (db: DatabaseConfig): PoolConfig => { + const config: PoolConfig = {}; + const iamAuthEnabled = Boolean(db.awsIamAuth?.enabled); + + if (iamAuthEnabled) { + // IAM auth supplies the password as a generated token, so the connection is + // driven by the discrete fields (or PG* env), never a connection string. + if (db.connectionString) { + console.warn( + '[postgres] awsIamAuth is enabled; ignoring connectionString (IAM mode uses the discrete host/port/user/database fields)', + ); + } + applyDiscreteFields(db, config, { includePassword: false }); + config.password = buildIamTokenProvider(db); + } else if (db.connectionString) { + if (hasDiscreteFields(db)) { + console.warn( + '[postgres] connectionString is set; ignoring the discrete host/port/user/password/database fields', + ); + } + config.connectionString = db.connectionString; + } else { + applyDiscreteFields(db, config, { includePassword: true }); + } + + applySsl(db, config, iamAuthEnabled); + applyPoolTuning(db, config); + return config; +}; + +const ensurePool = (): Pool => { + if (_pool) return _pool; + + const db = getDatabase(); + if (!hasConnectionConfig(db)) { + throw new Error( + 'Postgres connection is not configured (set connectionString, the host/port/user/password/database fields, or the PG* environment variables)', + ); + } + + _pool = new Pool(buildPoolConfig(db)); + // An idle client in the pool can emit 'error' (e.g. the backend dropped the + // connection). Without a listener node treats this as an uncaught exception + // and crashes the process; log it instead and let the pool recycle the client. + _pool.on('error', (err) => { + console.error('Postgres pool error on idle client:', err); + }); + return _pool; +}; + +/** + * Lazily resolves the pg Pool and runs any pending schema migrations exactly + * once per process. All adapter modules acquire the pool through this function + * so migrations complete before any query against `users` / `repos` / `pushes` + * is executed. + */ +export const connect = async (): Promise => { + const pool = ensurePool(); + if (!_bootstrapPromise) { + // `autoMigrate: false` supports deployments where the runtime role holds + // no DDL rights: migrations are applied out-of-band with elevated + // credentials (`npm run migrate:postgres:schema`) and startup only + // verifies the schema is current, failing fast when it is not. + const bootstrap = + getDatabase().autoMigrate === false ? assertMigrationsCurrent(pool) : runMigrations(pool); + _bootstrapPromise = bootstrap.catch((err) => { + // Reset so the next caller retries instead of being permanently latched + // onto a rejected promise. + _bootstrapPromise = null; + throw err; + }); + } + await _bootstrapPromise; + return pool; +}; + +/** + * Apply pending schema migrations regardless of the `autoMigrate` setting. + * Backs the `migrate:postgres:schema` npm script, which deployments running + * with `autoMigrate: false` use to apply DDL out-of-band with credentials + * that do hold DDL rights. + */ +export const applySchemaMigrations = async (): Promise => { + await runMigrations(ensurePool()); +}; + +/** + * Run `fn` inside a single transaction: every statement issued through the + * supplied client commits or rolls back together. Used where one logical + * update spans several statements, so a failure cannot leave partial state. + */ +export const withTransaction = async (fn: (client: PoolClient) => Promise): Promise => { + const pool = await connect(); + const client = await pool.connect(); + try { + await client.query('BEGIN'); + const result = await fn(client); + await client.query('COMMIT'); + return result; + } catch (err) { + try { + await client.query('ROLLBACK'); + } catch { + // the original error is the one worth surfacing + } + throw err; + } finally { + client.release(); + } +}; + +export const query = async ( + text: string, + params?: ReadonlyArray, +): Promise> => { + const pool = await connect(); + return pool.query(text, params as unknown[] | undefined); +}; + +/** + * Reset the pool and bootstrap latch — exported for test cleanup. + */ +export const resetConnection = async (): Promise => { + if (_pool) { + await _pool.end(); + _pool = null; + } + _bootstrapPromise = null; +}; + +/** + * Build an express-session Store backed by Postgres via `connect-pg-simple`. + * + * IMPORTANT: this function MUST NOT silently return undefined when Postgres is + * the active sink — that would cause express-session to fall back to its + * default in-memory store, which loses sessions on every restart and is unsafe + * in any multi-process deployment. Throw loudly instead. + */ +export const getSessionStore = (): Store => { + if (!hasConnectionConfig(getDatabase())) { + throw new Error( + 'Postgres connection is required for session storage (set connectionString, the host/port/user/password/database fields, or the PG* environment variables)', + ); + } + + const pool = ensurePool(); + const PgStore = connectPgSimple(session); + return new PgStore({ + pool, + tableName: 'session', + // The session table is owned by the versioned migration list (see + // schemaMigrations.ts) so all DDL lives in one place; letting the store + // create it would be a second, unversioned DDL path needing DDL rights + // at runtime even when autoMigrate is off. + createTableIfMissing: false, + }); +}; + +export const ensureSessionStoreReady = async (): Promise => { + // Run (or, with autoMigrate off, verify) migrations before probing the + // store: the session table is created by the migration runner, not the + // store itself. + await connect(); + const store = getSessionStore(); + + await new Promise((resolve, reject) => { + store.get('__git_proxy_session_startup_probe__', (err) => { + if (err) { + reject(err); + } else { + resolve(); + } + }); + }); + + const maybeClosableStore = store as Store & { close?: () => Promise }; + if (maybeClosableStore.close) { + await maybeClosableStore.close(); + } +}; diff --git a/src/db/postgres/index.ts b/src/db/postgres/index.ts new file mode 100644 index 000000000..8c903c555 --- /dev/null +++ b/src/db/postgres/index.ts @@ -0,0 +1,67 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import * as helper from './helper'; +import * as migrations from './migrations'; +import * as pushes from './pushes'; +import * as repo from './repo'; +import * as users from './users'; + +export const { getSessionStore, ensureSessionStoreReady } = helper; + +export const { + getPushes, + getPushesForUserProfile, + getRepoPushRollupsByCanonicalUrl, + writeAudit, + getPush, + deletePush, + authorise, + cancel, + reject, +} = pushes; + +export const { deriveCreatedAt, getAppliedMigrations, recordMigration, unrecordMigration } = + migrations; + +export const { + getRepos, + getRepo, + getRepoByUrl, + getRepoById, + createRepo, + updateRepo, + addUserCanPush, + addUserCanAuthorise, + removeUserCanPush, + removeUserCanAuthorise, + deleteRepo, +} = repo; + +export const { + findUser, + findUserByEmail, + findUserByGitAccount, + findUserByOIDC, + findUserBySSHKey, + getUsers, + createUser, + deleteUser, + updateUser, + addPublicKey, + removePublicKey, + getPublicKeys, +} = users; diff --git a/src/db/postgres/migrate.ts b/src/db/postgres/migrate.ts new file mode 100644 index 000000000..0924a84e2 --- /dev/null +++ b/src/db/postgres/migrate.ts @@ -0,0 +1,127 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Action } from '../../proxy/actions'; +import { Repo, User } from '../types'; + +/** + * A read-only view over a backend (mongo or fs) that data is migrated *from*. + * Implementations own their own connection and must be closed by the caller. + */ +export const DEFAULT_PUSH_BATCH_SIZE = 500; + +export interface MigrationSource { + getUsers(): Promise; + getRepos(): Promise; + /** + * Stream pushes in batches of at most `batchSize`. Production datasets can + * hold tens of thousands of large push documents, so sources must not + * require the whole table in memory at once. + */ + getPushBatches(batchSize: number): AsyncIterable; + close(): Promise; +} + +/** + * The subset of the Postgres adapter used to write migrated records. The + * adapter module satisfies this shape directly, so the CLI can pass it as-is. + */ +export interface MigrationDestination { + findUser(username: string): Promise; + findUserByEmail(email: string): Promise; + createUser(user: User): Promise; + getRepoByUrl(url: string): Promise; + createRepo(repo: Repo): Promise; + writeAudit(action: Action): Promise; +} + +export interface MigrationSummary { + users: { imported: number; skipped: number }; + repos: { imported: number; skipped: number }; + pushes: { imported: number }; +} + +export interface MigrateOptions { + /** Receives human-readable progress lines. Defaults to a no-op. */ + log?: (message: string) => void; + /** Maximum pushes fetched and written per batch. Defaults to 500. */ + pushBatchSize?: number; +} + +/** + * Copy users, repos and pushes from `source` into the Postgres `destination`. + * + * Idempotent and re-runnable: users and repos that already exist (matched by + * username/email and URL respectively) are skipped, and pushes are upserted by + * their stable string id. Record `_id`s are intentionally NOT carried over — + * Postgres assigns fresh UUIDs; push ids (TEXT) are preserved by the upsert. + */ +export const migrate = async ( + source: MigrationSource, + destination: MigrationDestination, + options: MigrateOptions = {}, +): Promise => { + const log = options.log ?? (() => undefined); + const summary: MigrationSummary = { + users: { imported: 0, skipped: 0 }, + repos: { imported: 0, skipped: 0 }, + pushes: { imported: 0 }, + }; + + const users = await source.getUsers(); + log(`Migrating ${users.length} user(s)...`); + for (const user of users) { + const existing = + (await destination.findUser(user.username)) || + (user.email ? await destination.findUserByEmail(user.email) : null); + if (existing) { + summary.users.skipped++; + continue; + } + // Legacy documents can lack optional fields the writers dereference: + // users synced from AD may have no email (the mail attribute is + // optional) or gitAccount. Default them like the mongo upsert path does. + await destination.createUser({ + ...user, + email: user.email ?? '', + gitAccount: user.gitAccount ?? '', + }); + summary.users.imported++; + } + + const repos = await source.getRepos(); + log(`Migrating ${repos.length} repo(s)...`); + for (const repo of repos) { + if (await destination.getRepoByUrl(repo.url)) { + summary.repos.skipped++; + continue; + } + await destination.createRepo(repo); + summary.repos.imported++; + } + + const batchSize = options.pushBatchSize ?? DEFAULT_PUSH_BATCH_SIZE; + log('Migrating pushes...'); + for await (const batch of source.getPushBatches(batchSize)) { + for (const push of batch) { + await destination.writeAudit(push); + summary.pushes.imported++; + } + log(` ${summary.pushes.imported} push(es) migrated`); + } + + return summary; +}; diff --git a/src/db/postgres/migrateFileSource.ts b/src/db/postgres/migrateFileSource.ts new file mode 100644 index 000000000..b4edcc4f1 --- /dev/null +++ b/src/db/postgres/migrateFileSource.ts @@ -0,0 +1,97 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import fs from 'fs'; +import path from 'path'; + +import Datastore from '@seald-io/nedb'; + +import { Action } from '../../proxy/actions'; +import { toClass } from '../helper'; +import { Repo, User } from '../types'; +import { MigrationSource } from './migrate'; + +// Where the `fs` sink keeps its NeDB datastores. +const DEFAULT_DATA_DIR = './.data/db'; + +/** + * Build a read-only {@link MigrationSource} backed by the NeDB datastores the + * `fs` sink writes. `dataDir` defaults to the location the sink uses + * (`./.data/db`). Record `_id`s are ignored by the Postgres writers, which + * assign fresh UUIDs. + */ +const DATASTORE_FILES = ['users.db', 'repos.db', 'pushes.db']; + +interface LazyStore { + store: Datastore; + ready: () => Promise; +} + +export const createFileSource = (dataDir: string = DEFAULT_DATA_DIR): MigrationSource => { + // Fail fast on a wrong path rather than reporting a legitimately empty + // backend: a missing directory, or one containing none of the fs sink's + // datastores, is a misconfiguration, while an existing-but-empty datastore + // is a real (empty) backend. + if (!fs.existsSync(dataDir)) { + throw new Error(`fs sink data directory does not exist: ${dataDir}`); + } + if (!DATASTORE_FILES.some((file) => fs.existsSync(path.join(dataDir, file)))) { + throw new Error(`No fs sink datastores (${DATASTORE_FILES.join(', ')}) found in: ${dataDir}`); + } + + // Loading is explicit (no autoload) so a corrupt datastore surfaces as a + // clear error instead of being silently treated as empty. + const load = (file: string): LazyStore => { + const filename = path.join(dataDir, file); + const store = new Datastore({ filename }); + let loading: Promise | undefined; + const ready = () => + (loading ??= store.loadDatabaseAsync().catch((err: unknown) => { + throw new Error( + `Failed to load ${filename}: ${err instanceof Error ? err.message : String(err)}`, + ); + })); + return { store, ready }; + }; + + const users = load('users.db'); + const repos = load('repos.db'); + const pushes = load('pushes.db'); + + const readAll = async ({ store, ready }: LazyStore, proto: object): Promise => { + await ready(); + const docs = await store.findAsync>({}); + return docs.map((doc) => toClass(doc, proto) as T); + }; + + // NeDB keeps the whole datastore in memory regardless, so batching here only + // shapes the write side to match the MigrationSource contract. + const getPushBatches = (batchSize: number): AsyncIterable => ({ + async *[Symbol.asyncIterator]() { + const all = await readAll(pushes, Action.prototype); + for (let i = 0; i < all.length; i += batchSize) { + yield all.slice(i, i + batchSize); + } + }, + }); + + return { + getUsers: () => readAll(users, User.prototype), + getRepos: () => readAll(repos, Repo.prototype), + getPushBatches, + close: () => Promise.resolve(), + }; +}; diff --git a/src/db/postgres/migrateMongoSource.ts b/src/db/postgres/migrateMongoSource.ts new file mode 100644 index 000000000..f8754c17f --- /dev/null +++ b/src/db/postgres/migrateMongoSource.ts @@ -0,0 +1,72 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { MongoClient, MongoClientOptions } from 'mongodb'; + +import { Action } from '../../proxy/actions'; +import { toClass } from '../helper'; +import { Repo, User } from '../types'; +import { MigrationSource } from './migrate'; + +/** + * Build a read-only {@link MigrationSource} backed by a MongoDB instance. + * + * The connection is explicit (not the configured sink) so the importer can read + * the legacy backend while the active sink points at the Postgres destination. + * The caller owns the lifecycle and must `close()` the source when done. + */ +export const createMongoSource = async ( + connectionString: string, + options: MongoClientOptions = {}, +): Promise => { + const client = new MongoClient(connectionString, options); + await client.connect(); + const db = client.db(); + + const readAll = async (collection: string, proto: object): Promise => { + const docs = await db.collection(collection).find().toArray(); + // toClass drops mongo class identity; the ObjectId `_id` it carries through + // is ignored by the Postgres writers, which assign fresh UUIDs. + return docs.map((doc) => toClass(doc, proto) as T); + }; + + // Pushes are streamed through a cursor rather than materialised: production + // tables can hold tens of thousands of documents that each carry a full + // diff, so a single toArray() would hold the entire table in memory. + const getPushBatches = (batchSize: number): AsyncIterable => ({ + async *[Symbol.asyncIterator]() { + const cursor = db.collection('pushes').find().batchSize(batchSize); + let batch: Action[] = []; + for await (const doc of cursor) { + batch.push(toClass(doc, Action.prototype) as Action); + if (batch.length >= batchSize) { + yield batch; + batch = []; + } + } + if (batch.length > 0) { + yield batch; + } + }, + }); + + return { + getUsers: () => readAll('users', User.prototype), + getRepos: () => readAll('repos', Repo.prototype), + getPushBatches, + close: () => client.close(), + }; +}; diff --git a/src/db/postgres/migrations.ts b/src/db/postgres/migrations.ts new file mode 100644 index 000000000..905e5d5db --- /dev/null +++ b/src/db/postgres/migrations.ts @@ -0,0 +1,37 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { query } from './helper'; + +/** + * PostgreSQL primary keys are random UUIDs (`gen_random_uuid()`), which carry no + * embedded creation time. Like the filesystem backend, this backend cannot + * recover a timestamp from an id, so callers fall back to their own default. + */ +export const deriveCreatedAt = (): string | undefined => undefined; + +export const getAppliedMigrations = async (): Promise => { + const result = await query<{ id: string }>(`SELECT id FROM migrations`); + return result.rows.map((row) => row.id); +}; + +export const recordMigration = async (id: string): Promise => { + await query(`INSERT INTO migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING`, [id]); +}; + +export const unrecordMigration = async (id: string): Promise => { + await query(`DELETE FROM migrations WHERE id = $1`, [id]); +}; diff --git a/src/db/postgres/pushes.ts b/src/db/postgres/pushes.ts new file mode 100644 index 000000000..97865295d --- /dev/null +++ b/src/db/postgres/pushes.ts @@ -0,0 +1,303 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { activityPrimaryStatusFromFlags } from '../../activity/activityPrimaryStatus'; +import { canonicalRemoteUrl } from '../../activity/canonicalRemoteUrl'; +import { Action } from '../../proxy/actions'; +import { CompletedAttestation, Rejection } from '../../proxy/processors/types'; +import { toClass } from '../helper'; +import { + emptyRepoActivityTabCounts, + PushQuery, + RepoActivityTabCounts, + RepoPushRollupsByCanonicalUrl, +} from '../types'; +import { query, withTransaction } from './helper'; + +const defaultPushQuery: Partial = { + error: false, + blocked: true, + allowPush: false, + authorised: false, + type: 'push', +}; + +// Columns that mirror Action fields used to filter `getPushes` results. +// Anything not in this map is ignored — the API only filters by these. +const FILTER_COLUMNS: Record = { + error: 'error', + blocked: 'blocked', + allowPush: 'allow_push', + authorised: 'authorised', + canceled: 'canceled', + rejected: 'rejected', + type: 'type', +}; + +const rowToAction = (row: { data: unknown }): Action => + toClass(row.data, Action.prototype) as Action; + +function bumpCount( + m: Map, + canonicalKey: string, + tab: keyof RepoActivityTabCounts, +): void { + if (!canonicalKey) { + return; + } + let row = m.get(canonicalKey); + if (!row) { + row = emptyRepoActivityTabCounts(); + m.set(canonicalKey, row); + } + row[tab] += 1; +} + +function bumpMaxTimestampMs( + m: Map, + canonicalKey: string, + timestamp: unknown, +): void { + if (!canonicalKey) { + return; + } + // `timestamp` is a BIGINT column, which node-postgres returns as a string. + // Anything else (null, undefined, empty) is not a usable timestamp. + const ts = + typeof timestamp === 'number' + ? timestamp + : typeof timestamp === 'string' && timestamp.trim() !== '' + ? Number(timestamp) + : NaN; + if (!Number.isFinite(ts)) { + return; + } + const prev = m.get(canonicalKey); + if (prev === undefined || ts > prev) { + m.set(canonicalKey, ts); + } +} + +/** + * Scan all push rows: tab counts and max timestamps per canonical remote URL + * (matches the Activity UI). The URL lives inside the `data` JSONB payload, and + * canonicalization happens in Node so the result matches the mongo and fs + * backends exactly. + */ +export const getRepoPushRollupsByCanonicalUrl = + async (): Promise => { + const result = await query<{ + url: string | null; + error: boolean; + rejected: boolean; + canceled: boolean; + authorised: boolean; + blocked: boolean; + allow_push: boolean; + timestamp: string | number | null; + }>( + `SELECT data->>'url' AS url, error, rejected, canceled, authorised, blocked, + allow_push, timestamp + FROM pushes + WHERE type = 'push'`, + ); + + const tabCounts = new Map(); + const latestPendingReviewAtMs = new Map(); + const latestPushAtMs = new Map(); + + for (const row of result.rows) { + const url = typeof row.url === 'string' ? row.url : ''; + const key = canonicalRemoteUrl(url); + if (!key) { + continue; + } + const tab = activityPrimaryStatusFromFlags({ + error: row.error === true, + rejected: row.rejected === true, + canceled: row.canceled === true, + authorised: row.authorised === true, + blocked: row.blocked === true, + allowPush: row.allow_push === true, + }); + bumpCount(tabCounts, key, tab); + bumpMaxTimestampMs(latestPushAtMs, key, row.timestamp); + if (tab === 'pending') { + bumpMaxTimestampMs(latestPendingReviewAtMs, key, row.timestamp); + } + } + + return { tabCounts, latestPendingReviewAtMs, latestPushAtMs }; + }; + +/** + * Pushes shown on a user profile: those the user made (any known email variant) + * plus those they reviewed. Mirrors `buildUserProfilePushFilter`, which the + * mongo and fs backends feed to their query engines; the reviewer match is + * case-insensitive on the exact username. + */ +export const getPushesForUserProfile = async ( + emailVariants: string[], + profileUsername: string, +): Promise => { + const reviewerClause = `lower(data->'attestation'->'reviewer'->>'username') = lower($1)`; + const values: unknown[] = [profileUsername]; + let predicate = reviewerClause; + + if (emailVariants.length > 0) { + values.push(emailVariants); + predicate = `((data->>'userEmail') = ANY($${values.length}::text[]) OR ${reviewerClause})`; + } + + const result = await query<{ data: unknown }>( + `SELECT data - 'steps' AS data FROM pushes WHERE type = 'push' AND ${predicate} ORDER BY timestamp DESC`, + values, + ); + return result.rows.map(rowToAction); +}; + +// List queries drop `steps` from the returned document: it holds the full diff +// (largest part of a push row) and the mongo backend's list projection excludes +// it as well. The push-detail path (`getPush`) still returns the whole document. +export const getPushes = async (q: Partial = defaultPushQuery): Promise => { + const clauses: string[] = []; + const values: unknown[] = []; + for (const [key, value] of Object.entries(q)) { + const column = FILTER_COLUMNS[key]; + if (!column || value === undefined) continue; + values.push(value); + clauses.push(`${column} = $${values.length}`); + } + + const where = clauses.length > 0 ? `WHERE ${clauses.join(' AND ')}` : ''; + const result = await query<{ data: unknown }>( + `SELECT data - 'steps' AS data FROM pushes ${where} ORDER BY timestamp DESC`, + values, + ); + return result.rows.map(rowToAction); +}; + +export const getPush = async (id: string): Promise => { + const result = await query<{ data: unknown }>(`SELECT data FROM pushes WHERE id = $1`, [id]); + if (result.rowCount === 0) return null; + return rowToAction(result.rows[0]); +}; + +export const deletePush = async (id: string): Promise => { + await query(`DELETE FROM pushes WHERE id = $1`, [id]); +}; + +const buildAuditUpsert = (action: Action): { text: string; values: unknown[] } => { + if (typeof action.id !== 'string') { + throw new Error('Invalid id'); + } + + // Round-trip through JSON to drop class identity / mongo-specific _id fields + // before persisting (mirrors mongo's `JSON.parse(JSON.stringify(action))`). + const data = JSON.parse(JSON.stringify(action)); + delete data._id; + + return { + text: `INSERT INTO pushes ( + id, timestamp, type, error, blocked, allow_push, + authorised, canceled, rejected, data + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10::jsonb) + ON CONFLICT (id) DO UPDATE SET + timestamp = EXCLUDED.timestamp, + type = EXCLUDED.type, + error = EXCLUDED.error, + blocked = EXCLUDED.blocked, + allow_push = EXCLUDED.allow_push, + authorised = EXCLUDED.authorised, + canceled = EXCLUDED.canceled, + rejected = EXCLUDED.rejected, + data = EXCLUDED.data`, + values: [ + action.id, + action.timestamp ?? Date.now(), + action.type ?? null, + action.error ?? false, + action.blocked ?? false, + action.allowPush ?? false, + action.authorised ?? false, + action.canceled ?? false, + action.rejected ?? false, + JSON.stringify(data), + ], + }; +}; + +export const writeAudit = async (action: Action): Promise => { + const { text, values } = buildAuditUpsert(action); + await query(text, values); +}; + +/** + * Load a push, apply `mutate`, and persist the result atomically. The row is + * read with `FOR UPDATE` inside a transaction, so two concurrent decisions on + * the same push (or a step-result write from the proxy racing a reviewer's + * decision) serialise instead of the later write silently discarding the + * earlier one. + */ +const mutatePush = async (id: string, mutate: (action: Action) => void): Promise => + withTransaction(async (client) => { + const result = await client.query<{ data: unknown }>( + `SELECT data FROM pushes WHERE id = $1 FOR UPDATE`, + [id], + ); + if (result.rowCount === 0) { + throw new Error(`push ${id} not found`); + } + const action = rowToAction(result.rows[0]); + mutate(action); + const { text, values } = buildAuditUpsert(action); + await client.query(text, values); + }); + +export const authorise = async ( + id: string, + attestation?: CompletedAttestation, +): Promise<{ message: string }> => { + await mutatePush(id, (action) => { + action.authorised = true; + action.canceled = false; + action.rejected = false; + action.attestation = attestation; + }); + return { message: `authorised ${id}` }; +}; + +export const reject = async (id: string, rejection: Rejection): Promise<{ message: string }> => { + await mutatePush(id, (action) => { + action.authorised = false; + action.canceled = false; + action.rejected = true; + // Preserve the existing rejection-payload shape used by the fs/mongo + // backends — the issue calls this out explicitly as a must-fix. + action.rejection = rejection; + }); + return { message: `reject ${id}` }; +}; + +export const cancel = async (id: string): Promise<{ message: string }> => { + await mutatePush(id, (action) => { + action.authorised = false; + action.canceled = true; + action.rejected = false; + }); + return { message: `canceled ${id}` }; +}; diff --git a/src/db/postgres/repo.ts b/src/db/postgres/repo.ts new file mode 100644 index 000000000..57c5d89f3 --- /dev/null +++ b/src/db/postgres/repo.ts @@ -0,0 +1,259 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { PoolClient } from 'pg'; + +import { Repo, RepoQuery } from '../types'; +import { query, withTransaction } from './helper'; + +interface RepoRow { + _id: string; + project: string; + name: string; + url: string; + can_push: string[] | null; + can_authorise: string[] | null; + date_created: string | null; + last_modified: string | null; +} + +const rowToRepo = (row: RepoRow): Repo => + new Repo( + row.project, + row.name, + row.url, + { + canPush: row.can_push ?? [], + canAuthorise: row.can_authorise ?? [], + }, + row._id, + row.date_created ?? undefined, + row.last_modified ?? undefined, + ); + +// Reconstruct the `canPush` / `canAuthorise` arrays from the normalised +// repo_users join table. `ORDER BY` keeps the arrays deterministic, and the +// `coalesce(..., '{}')` makes a repo with no members come back as empty arrays +// rather than null, matching the mongo/NeDB backends. +const SELECT_REPOS = ` + SELECT r._id, r.project, r.name, r.url, r.date_created, r.last_modified, + coalesce( + array_agg(ru.username ORDER BY ru.username) FILTER (WHERE ru.role = 'canPush'), + '{}' + ) AS can_push, + coalesce( + array_agg(ru.username ORDER BY ru.username) FILTER (WHERE ru.role = 'canAuthorise'), + '{}' + ) AS can_authorise + FROM repos r + LEFT JOIN repo_users ru ON ru.repo_id = r._id`; + +const GROUP_BY = 'GROUP BY r._id'; + +export const getRepos = async (q: Partial = {}): Promise => { + const clauses: string[] = []; + const values: unknown[] = []; + if (q.name) { + values.push(q.name.toLowerCase()); + clauses.push(`r.name = $${values.length}`); + } + if (q.project !== undefined) { + values.push(q.project); + clauses.push(`r.project = $${values.length}`); + } + if (q.url) { + values.push(q.url); + clauses.push(`r.url = $${values.length}`); + } + + const where = clauses.length > 0 ? `WHERE ${clauses.join(' AND ')}` : ''; + const result = await query(`${SELECT_REPOS} ${where} ${GROUP_BY}`, values); + return result.rows.map(rowToRepo); +}; + +export const getRepo = async (name: string): Promise => { + const result = await query(`${SELECT_REPOS} WHERE r.name = $1 ${GROUP_BY}`, [ + name.toLowerCase(), + ]); + return result.rowCount === 0 ? null : rowToRepo(result.rows[0]); +}; + +export const getRepoByUrl = async (url: string): Promise => { + const result = await query(`${SELECT_REPOS} WHERE r.url = $1 ${GROUP_BY}`, [url]); + return result.rowCount === 0 ? null : rowToRepo(result.rows[0]); +}; + +export const getRepoById = async (_id: string): Promise => { + const result = await query(`${SELECT_REPOS} WHERE r._id = $1 ${GROUP_BY}`, [_id]); + return result.rowCount === 0 ? null : rowToRepo(result.rows[0]); +}; + +const addUserToRole = async ( + _id: string, + user: string, + role: 'canPush' | 'canAuthorise', +): Promise => { + await query( + `INSERT INTO repo_users (repo_id, username, role) + VALUES ($1, $2, $3) + ON CONFLICT DO NOTHING`, + [_id, user.toLowerCase(), role], + ); + await query(`UPDATE repos SET last_modified = $2 WHERE _id = $1`, [ + _id, + new Date().toISOString(), + ]); +}; + +const removeUserFromRole = async ( + _id: string, + user: string, + role: 'canPush' | 'canAuthorise', +): Promise => { + await query(`DELETE FROM repo_users WHERE repo_id = $1 AND username = $2 AND role = $3`, [ + _id, + user.toLowerCase(), + role, + ]); + await query(`UPDATE repos SET last_modified = $2 WHERE _id = $1`, [ + _id, + new Date().toISOString(), + ]); +}; + +// Insert one role's usernames as a single statement. Lowercased to match +// addUserToRole; ON CONFLICT collapses duplicates (case-only ones included). +const insertRoleRows = async ( + client: PoolClient, + _id: string, + role: 'canPush' | 'canAuthorise', + usernames: string[], +): Promise => { + if (usernames.length === 0) return; + await client.query( + `INSERT INTO repo_users (repo_id, username, role) + SELECT $1, lower(u.username), $2 FROM unnest($3::text[]) AS u(username) + ON CONFLICT DO NOTHING`, + [_id, role, usernames], + ); +}; + +export const createRepo = async (repo: Repo): Promise => { + const users = repo.users ?? { canPush: [], canAuthorise: [] }; + const now = new Date().toISOString(); + if (!repo.dateCreated) repo.dateCreated = now; + if (!repo.lastModified) repo.lastModified = now; + + // One transaction: the repo row and any permissions supplied at creation + // land together or not at all. A crash partway must not leave a repo behind + // with empty canPush/canAuthorise, since those arrays gate pushing and + // approving. + const _id = await withTransaction(async (client) => { + const result = await client.query<{ _id: string }>( + `INSERT INTO repos (project, name, url, date_created, last_modified) + VALUES ($1, $2, $3, $4, $5) + RETURNING _id`, + [repo.project ?? '', repo.name, repo.url, repo.dateCreated, repo.lastModified], + ); + const newId = result.rows[0]._id; + await insertRoleRows(client, newId, 'canPush', users.canPush ?? []); + await insertRoleRows(client, newId, 'canAuthorise', users.canAuthorise ?? []); + return newId; + }); + + repo._id = _id; + repo.users = users; + return repo; +}; + +/** + * Apply a partial update to a repo row. Only the supplied fields are written, + * matching mongo's `$set` / `$unset` behaviour: a field explicitly set to + * `undefined` is reset to the column default. + * + * Permissions live in the `repo_users` join table rather than a column, so a + * supplied `users` object replaces that repo's rows wholesale. + */ +export const updateRepo = async (repo: Partial): Promise => { + const { _id, users, ...fields } = repo; + if (!_id) { + throw new Error('updateRepo requires a repo _id'); + } + + const COLUMNS: Record = { + project: 'project', + name: 'name', + url: 'url', + dateCreated: 'date_created', + lastModified: 'last_modified', + }; + + const sets: string[] = []; + const values: unknown[] = []; + for (const [key, value] of Object.entries(fields)) { + const column = COLUMNS[key]; + if (!column) continue; + if (value === undefined) { + sets.push(`${column} = DEFAULT`); + continue; + } + values.push(value); + sets.push(`${column} = $${values.length}`); + } + + if (sets.length === 0 && users === undefined) { + throw new Error('updateRepo requires at least one field to update'); + } + + // One transaction for the whole update: the row change, the permission + // replacement and the last_modified bump land together or not at all, so a + // failure partway cannot leave a repo without its roles. + await withTransaction(async (client) => { + if (sets.length > 0) { + await client.query(`UPDATE repos SET ${sets.join(', ')} WHERE _id = $${values.length + 1}`, [ + ...values, + _id, + ]); + } + + if (users !== undefined) { + await client.query(`DELETE FROM repo_users WHERE repo_id = $1`, [_id]); + await insertRoleRows(client, _id, 'canPush', users.canPush ?? []); + await insertRoleRows(client, _id, 'canAuthorise', users.canAuthorise ?? []); + await client.query(`UPDATE repos SET last_modified = $2 WHERE _id = $1`, [ + _id, + new Date().toISOString(), + ]); + } + }); +}; + +export const addUserCanPush = (_id: string, user: string): Promise => + addUserToRole(_id, user, 'canPush'); + +export const addUserCanAuthorise = (_id: string, user: string): Promise => + addUserToRole(_id, user, 'canAuthorise'); + +export const removeUserCanPush = (_id: string, user: string): Promise => + removeUserFromRole(_id, user, 'canPush'); + +export const removeUserCanAuthorise = (_id: string, user: string): Promise => + removeUserFromRole(_id, user, 'canAuthorise'); + +export const deleteRepo = async (_id: string): Promise => { + // repo_users rows are removed by the ON DELETE CASCADE foreign key. + await query(`DELETE FROM repos WHERE _id = $1`, [_id]); +}; diff --git a/src/db/postgres/schemaMigrations.ts b/src/db/postgres/schemaMigrations.ts new file mode 100644 index 000000000..7d1944a94 --- /dev/null +++ b/src/db/postgres/schemaMigrations.ts @@ -0,0 +1,288 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Pool } from 'pg'; + +/** + * A single, immutable schema change. Append new migrations with the next + * `version`; never edit or reorder entries that have already shipped, since + * deployed databases record which versions they have applied. + */ +export interface Migration { + version: number; + name: string; + sql: string; +} + +/** + * Ordered, append-only list of schema migrations. + * + * Version 1 is the initial schema. Because every statement uses + * `CREATE TABLE/INDEX IF NOT EXISTS`, databases that were already bootstrapped + * by the pre-migration code adopt the runner transparently: the statements are + * no-ops and version 1 is simply recorded as applied. + */ +export const MIGRATIONS: Migration[] = [ + { + version: 1, + name: 'initial_schema', + sql: ` + CREATE TABLE IF NOT EXISTS users ( + _id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + username TEXT NOT NULL UNIQUE, + email TEXT NOT NULL UNIQUE, + password TEXT, + git_account TEXT NOT NULL, + admin BOOLEAN NOT NULL DEFAULT FALSE, + oidc_id TEXT UNIQUE, + display_name TEXT, + title TEXT + ); + + CREATE TABLE IF NOT EXISTS repos ( + _id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + project TEXT NOT NULL DEFAULT '', + name TEXT NOT NULL, + url TEXT NOT NULL UNIQUE, + users JSONB NOT NULL DEFAULT '{"canPush":[],"canAuthorise":[]}'::jsonb, + date_created TEXT, + last_modified TEXT + ); + ALTER TABLE repos ADD COLUMN IF NOT EXISTS date_created TEXT; + ALTER TABLE repos ADD COLUMN IF NOT EXISTS last_modified TEXT; + CREATE INDEX IF NOT EXISTS repos_name_idx ON repos (name); + + CREATE TABLE IF NOT EXISTS pushes ( + id TEXT PRIMARY KEY, + timestamp BIGINT NOT NULL, + type TEXT, + error BOOLEAN NOT NULL DEFAULT FALSE, + blocked BOOLEAN NOT NULL DEFAULT FALSE, + allow_push BOOLEAN NOT NULL DEFAULT FALSE, + authorised BOOLEAN NOT NULL DEFAULT FALSE, + canceled BOOLEAN NOT NULL DEFAULT FALSE, + rejected BOOLEAN NOT NULL DEFAULT FALSE, + data JSONB NOT NULL + ); + CREATE INDEX IF NOT EXISTS pushes_timestamp_idx ON pushes (timestamp DESC); +`, + }, + { + version: 2, + name: 'user_public_keys_and_optional_email', + sql: ` + ALTER TABLE users ADD COLUMN IF NOT EXISTS public_keys JSONB NOT NULL DEFAULT '[]'::jsonb; + ALTER TABLE users ALTER COLUMN email DROP NOT NULL; + ALTER TABLE users DROP CONSTRAINT IF EXISTS users_email_key; + -- Email uniqueness is best-effort, like the mongo/fs backends: a real + -- address can only be claimed once, but any number of users may have no + -- email (the AD "mail" attribute is optional, for instance). + CREATE UNIQUE INDEX IF NOT EXISTS users_email_unique + ON users (email) WHERE email IS NOT NULL AND email <> ''; +`, + }, + { + version: 3, + name: 'migration_bookkeeping', + sql: ` + -- Bookkeeping for the cross-backend migration framework in src/db/migrations. + -- That framework records logical migrations by string id through the Sink + -- hooks; this table is its postgres storage, and is separate from the + -- schema_migrations table that versions the DDL below. + CREATE TABLE IF NOT EXISTS migrations ( + id TEXT PRIMARY KEY + ); +`, + }, + { + version: 4, + name: 'repo_users_table', + sql: ` + CREATE TABLE IF NOT EXISTS repo_users ( + repo_id UUID NOT NULL REFERENCES repos(_id) ON DELETE CASCADE, + username TEXT NOT NULL, + role TEXT NOT NULL CHECK (role IN ('canPush', 'canAuthorise')), + PRIMARY KEY (repo_id, username, role) + ); + CREATE INDEX IF NOT EXISTS repo_users_repo_id_idx ON repo_users (repo_id); + + -- Backfill the normalised table from the existing JSONB permissions. The + -- legacy repos.users column is dropped in a later migration once the adapter + -- reads and writes repo_users instead. + -- Usernames are lowercased to match the runtime writers (addUserToRole and + -- friends lowercase on insert), so legacy mixed-case entries stay + -- retrievable; ON CONFLICT collapses any case-only duplicates. + INSERT INTO repo_users (repo_id, username, role) + SELECT r._id, lower(elem.username), 'canPush' + FROM repos r, + jsonb_array_elements_text(coalesce(r.users->'canPush', '[]'::jsonb)) AS elem(username) + ON CONFLICT DO NOTHING; + + INSERT INTO repo_users (repo_id, username, role) + SELECT r._id, lower(elem.username), 'canAuthorise' + FROM repos r, + jsonb_array_elements_text(coalesce(r.users->'canAuthorise', '[]'::jsonb)) AS elem(username) + ON CONFLICT DO NOTHING; +`, + }, + { + version: 5, + name: 'drop_repos_users_jsonb', + // The repo adapter now reads and writes permissions via repo_users, so the + // legacy JSONB column (backfilled in migration 4) is no longer used. + sql: `ALTER TABLE repos DROP COLUMN IF EXISTS users;`, + }, + { + version: 6, + name: 'pushes_hot_path_indexes', + sql: ` + -- Covering index for the repo activity rollup: the scan becomes index-only + -- and never detoasts the large push JSONB documents. + CREATE INDEX IF NOT EXISTS pushes_rollup_idx + ON pushes ((data->>'url'), timestamp) + INCLUDE (error, rejected, canceled, authorised, blocked, allow_push) + WHERE type = 'push'; + + -- Matches the default dashboard query for pushes pending review. + CREATE INDEX IF NOT EXISTS pushes_pending_idx + ON pushes (timestamp DESC) + WHERE type = 'push' AND blocked AND NOT error AND NOT authorised AND NOT allow_push; + + -- User profile lookups filter on JSONB expressions; index both predicates. + CREATE INDEX IF NOT EXISTS pushes_user_email_idx + ON pushes ((data->>'userEmail')) + WHERE type = 'push'; + CREATE INDEX IF NOT EXISTS pushes_reviewer_idx + ON pushes ((lower(data->'attestation'->'reviewer'->>'username'))) + WHERE type = 'push'; +`, + }, + { + version: 7, + name: 'session_table', + // The express-session table, previously created by connect-pg-simple's + // createTableIfMissing outside the versioned migration list. Owning it + // here keeps all DDL in one place, so a role without DDL rights can run + // with autoMigrate off. The definition matches connect-pg-simple's + // table.sql, and IF NOT EXISTS adopts databases where the store already + // created it. + sql: ` + CREATE TABLE IF NOT EXISTS "session" ( + "sid" varchar NOT NULL COLLATE "default", + "sess" json NOT NULL, + "expire" timestamp(6) NOT NULL, + CONSTRAINT "session_pkey" PRIMARY KEY ("sid") NOT DEFERRABLE INITIALLY IMMEDIATE + ); + CREATE INDEX IF NOT EXISTS "IDX_session_expire" ON "session" ("expire"); +`, + }, +]; + +const SCHEMA_MIGRATIONS_TABLE_SQL = ` + CREATE TABLE IF NOT EXISTS schema_migrations ( + version INTEGER PRIMARY KEY, + name TEXT NOT NULL, + applied_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); +`; + +// Fixed, arbitrary advisory-lock key. Serialises migration runs across +// concurrently starting processes so each migration is applied exactly once. +const MIGRATION_ADVISORY_LOCK_KEY = 4815162342; + +/** + * Apply any not-yet-applied migrations in version order, inside a single + * transaction guarded by a transaction-scoped advisory lock. + * + * Safe to call on every process start: already-applied migrations are skipped, + * and concurrent callers block on the lock rather than racing. The lock is + * acquired before the `schema_migrations` table is touched so two processes + * booting against a brand-new database cannot both seed version 1. + * + * NOTE: all pending migrations run in one transaction, so a future statement + * that cannot run transactionally (e.g. `CREATE INDEX CONCURRENTLY`) will need + * dedicated handling — not required for the current schema. + */ +export const runMigrations = async (pool: Pool): Promise => { + const client = await pool.connect(); + try { + await client.query('BEGIN'); + // Transaction-scoped lock; auto-released on COMMIT/ROLLBACK. + await client.query('SELECT pg_advisory_xact_lock($1)', [MIGRATION_ADVISORY_LOCK_KEY]); + await client.query(SCHEMA_MIGRATIONS_TABLE_SQL); + + const { rows } = await client.query<{ version: number }>( + 'SELECT version FROM schema_migrations', + ); + const applied = new Set(rows.map((row) => row.version)); + + const pending = [...MIGRATIONS] + .sort((a, b) => a.version - b.version) + .filter((migration) => !applied.has(migration.version)); + + for (const migration of pending) { + await client.query(migration.sql); + await client.query('INSERT INTO schema_migrations (version, name) VALUES ($1, $2)', [ + migration.version, + migration.name, + ]); + } + + await client.query('COMMIT'); + } catch (err) { + try { + await client.query('ROLLBACK'); + } catch { + // best-effort rollback; the original error is rethrown below + } + throw err; + } finally { + client.release(); + } +}; + +/** + * Verify that every known migration has been applied, without issuing any DDL. + * Used at startup when `autoMigrate` is disabled: the operator applies + * migrations out-of-band (`npm run migrate:postgres:schema`, run with + * DDL-capable credentials) and the runtime role only needs DML rights. + * Throws a descriptive error naming the pending versions. + */ +export const assertMigrationsCurrent = async (pool: Pool): Promise => { + // to_regclass returns null when the table does not exist, so a brand-new + // database is reported as "everything pending" rather than a query error. + const tableCheck = await pool.query<{ table_oid: string | null }>( + `SELECT to_regclass('schema_migrations') AS table_oid`, + ); + + const applied = new Set(); + if (tableCheck.rows[0]?.table_oid) { + const { rows } = await pool.query<{ version: number }>('SELECT version FROM schema_migrations'); + for (const row of rows) { + applied.add(row.version); + } + } + + const pending = MIGRATIONS.filter((migration) => !applied.has(migration.version)); + if (pending.length > 0) { + const versions = pending.map((m) => `${m.version} (${m.name})`).join(', '); + throw new Error( + `postgres schema is out of date and autoMigrate is disabled; pending migrations: ${versions}. ` + + 'Apply them with DDL-capable credentials via `npm run migrate:postgres:schema` ' + + '(or temporarily enable autoMigrate), then restart.', + ); + } +}; diff --git a/src/db/postgres/users.ts b/src/db/postgres/users.ts new file mode 100644 index 000000000..9d0f3d273 --- /dev/null +++ b/src/db/postgres/users.ts @@ -0,0 +1,274 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { PublicKeyRecord, User, UserQuery } from '../types'; +import { DuplicateSSHKeyError } from '../../errors/DatabaseErrors'; +import { query, withTransaction } from './helper'; + +interface UserRow { + _id: string; + username: string; + email: string | null; + password: string | null; + git_account: string; + admin: boolean; + oidc_id: string | null; + public_keys: PublicKeyRecord[] | null; + display_name: string | null; + title: string | null; +} + +const rowToUser = (row: UserRow): User => { + const user = new User( + row.username, + row.password ?? '', + row.git_account, + row.email ?? '', + row.admin, + row.oidc_id, + row.public_keys ?? [], + row._id, + ); + user.password = row.password; + user.displayName = row.display_name; + user.title = row.title; + return user; +}; + +const SELECT_COLUMNS = + '_id, username, email, password, git_account, admin, oidc_id, public_keys, display_name, title'; + +export const findUser = async (username: string): Promise => { + const result = await query(`SELECT ${SELECT_COLUMNS} FROM users WHERE username = $1`, [ + username.toLowerCase(), + ]); + return result.rowCount === 0 ? null : rowToUser(result.rows[0]); +}; + +export const findUserByEmail = async (email: string): Promise => { + const result = await query(`SELECT ${SELECT_COLUMNS} FROM users WHERE email = $1`, [ + email.toLowerCase(), + ]); + return result.rowCount === 0 ? null : rowToUser(result.rows[0]); +}; + +export const findUserByGitAccount = async (gitAccount: string): Promise => { + const result = await query( + `SELECT ${SELECT_COLUMNS} FROM users WHERE git_account = $1`, + [gitAccount.toLowerCase()], + ); + return result.rowCount === 0 ? null : rowToUser(result.rows[0]); +}; + +export const findUserByOIDC = async (oidcId: string): Promise => { + const result = await query(`SELECT ${SELECT_COLUMNS} FROM users WHERE oidc_id = $1`, [ + oidcId, + ]); + return result.rowCount === 0 ? null : rowToUser(result.rows[0]); +}; + +export const getUsers = async (q: Partial = {}): Promise => { + const clauses: string[] = []; + const values: unknown[] = []; + if (q.username) { + values.push(q.username.toLowerCase()); + clauses.push(`username = $${values.length}`); + } + if (q.email) { + values.push(q.email.toLowerCase()); + clauses.push(`email = $${values.length}`); + } + + const where = clauses.length > 0 ? `WHERE ${clauses.join(' AND ')}` : ''; + // Match mongo's `.project({ password: 0 })` — omit password from list results. + const result = await query( + `SELECT _id, username, email, NULL::text AS password, git_account, admin, oidc_id, public_keys, display_name, title + FROM users ${where}`, + values, + ); + return result.rows.map(rowToUser); +}; + +export const createUser = async (user: User): Promise => { + await query( + `INSERT INTO users (username, email, password, git_account, admin, oidc_id, public_keys, display_name, title) + VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb, $8, $9)`, + [ + user.username.toLowerCase(), + user.email.toLowerCase(), + user.password ?? null, + user.gitAccount, + user.admin, + user.oidcId ?? null, + JSON.stringify(user.publicKeys ?? []), + user.displayName ?? null, + user.title ?? null, + ], + ); +}; + +export const deleteUser = async (username: string): Promise => { + await query(`DELETE FROM users WHERE username = $1`, [username.toLowerCase()]); +}; + +/** + * Update an existing user, or insert a new one if no matching row exists. + * + * Mirrors the mongo adapter's upsert semantics: partial updates are merged + * onto an existing row (only supplied fields are written), and a missing row + * is created. Identity is by `_id` when provided, otherwise by `username`. + */ +export const updateUser = async (user: Partial): Promise => { + const username = user.username?.toLowerCase(); + const email = user.email?.toLowerCase(); + + // Track the supplied columns so both branches only ever write the fields + // the caller patched. + const columns: string[] = []; + const values: unknown[] = []; + const set = (column: string, value: unknown) => { + columns.push(column); + values.push(value); + }; + + if (username !== undefined) set('username', username); + if (email !== undefined) set('email', email); + if (user.password !== undefined) set('password', user.password); + if (user.gitAccount !== undefined) set('git_account', user.gitAccount); + if (user.admin !== undefined) set('admin', user.admin); + if (user.oidcId !== undefined) set('oidc_id', user.oidcId); + if (user.publicKeys !== undefined) set('public_keys', JSON.stringify(user.publicKeys)); + if (user.displayName !== undefined) set('display_name', user.displayName); + if (user.title !== undefined) set('title', user.title); + + // An empty SET list would be a SQL syntax error, so fail loudly rather than + // let callers (or future handlers copying this builder) hit that. + if (columns.length === 0) { + throw new Error('updateUser requires at least one field to update'); + } + + if (user._id) { + const sets = columns.map((column, i) => `${column} = $${i + 1}`); + values.push(user._id); + await query(`UPDATE users SET ${sets.join(', ')} WHERE _id = $${values.length}`, values); + return; + } + + if (!username) { + throw new Error('updateUser requires either _id or username'); + } + + // Upsert by username when no _id is supplied, matching mongo's behaviour. + // A single atomic statement (rather than UPDATE-then-INSERT) so a + // concurrent insert of the same username can't drop the update; on + // conflict only the supplied fields are merged onto the existing row. + const assignments = columns.map((column) => `${column} = EXCLUDED.${column}`); + await query( + `INSERT INTO users (username, email, password, git_account, admin, oidc_id, public_keys, display_name, title) + VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb, $8, $9) + ON CONFLICT (username) DO UPDATE SET ${assignments.join(', ')}`, + [ + username, + email ?? null, + user.password ?? null, + user.gitAccount ?? '', + user.admin ?? false, + user.oidcId ?? null, + JSON.stringify(user.publicKeys ?? []), + user.displayName ?? null, + user.title ?? null, + ], + ); +}; + +export const findUserBySSHKey = async (sshKey: string): Promise => { + // JSONB containment: matches any element of public_keys with this exact key, + // equivalent to mongo's `{ 'publicKeys.key': sshKey }`. + const result = await query( + `SELECT ${SELECT_COLUMNS} FROM users WHERE public_keys @> $1::jsonb`, + [JSON.stringify([{ key: sshKey }])], + ); + return result.rowCount === 0 ? null : rowToUser(result.rows[0]); +}; + +export const addPublicKey = async (username: string, publicKey: PublicKeyRecord): Promise => { + await withTransaction(async (client) => { + // Key uniqueness spans elements of a JSONB array, which no unique + // constraint can enforce, so concurrent adds of the same key are + // serialised on a transaction-scoped advisory lock derived from the key + // text: the loser waits here and then sees the winner's row in the + // duplicate check below. + await client.query(`SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, [publicKey.key]); + + const existing = await client.query( + `SELECT ${SELECT_COLUMNS} FROM users WHERE public_keys @> $1::jsonb`, + [JSON.stringify([{ key: publicKey.key }])], + ); + const existingUser = existing.rowCount === 0 ? null : rowToUser(existing.rows[0]); + if (existingUser && existingUser.username.toLowerCase() !== username.toLowerCase()) { + throw new DuplicateSSHKeyError(existingUser.username); + } + + // Lock the target row so a concurrent add of a different key for the same + // user cannot interleave with the duplicate-fingerprint check. + const found = await client.query( + `SELECT ${SELECT_COLUMNS} FROM users WHERE username = $1 FOR UPDATE`, + [username.toLowerCase()], + ); + if (found.rowCount === 0) { + throw new Error('User not found'); + } + const user = rowToUser(found.rows[0]); + + const keyExists = user.publicKeys?.some( + (k) => k.key === publicKey.key || (k.fingerprint && k.fingerprint === publicKey.fingerprint), + ); + if (keyExists) { + throw new Error('SSH key already exists'); + } + + await client.query( + `UPDATE users SET public_keys = public_keys || $2::jsonb WHERE username = $1`, + [username.toLowerCase(), JSON.stringify([publicKey])], + ); + }); +}; + +export const removePublicKey = async (username: string, fingerprint: string): Promise => { + // Filter the matching key out of the JSONB array; like mongo's `$pull`, this + // is a no-op when the user or fingerprint does not exist. + await query( + `UPDATE users + SET public_keys = coalesce( + ( + SELECT jsonb_agg(k) + FROM jsonb_array_elements(public_keys) AS k + WHERE (k->>'fingerprint') IS DISTINCT FROM $2 + ), + '[]'::jsonb + ) + WHERE username = $1`, + [username.toLowerCase(), fingerprint], + ); +}; + +export const getPublicKeys = async (username: string): Promise => { + const user = await findUser(username); + if (!user) { + throw new Error('User not found'); + } + return user.publicKeys || []; +}; diff --git a/src/db/types.ts b/src/db/types.ts index 10f35ba5d..99f185c9c 100644 --- a/src/db/types.ts +++ b/src/db/types.ts @@ -16,6 +16,7 @@ import { Action } from '../proxy/actions/Action'; import MongoDBStore from 'connect-mongo'; +import { Store } from 'express-session'; import { CompletedAttestation, Rejection } from '../proxy/processors/types'; export type PushQuery = { @@ -169,7 +170,8 @@ export interface PublicUser { } export interface Sink { - getSessionStore: () => MongoDBStore | undefined; + getSessionStore: () => MongoDBStore | Store | undefined; + ensureSessionStoreReady?: () => Promise; getRepoPushRollupsByCanonicalUrl: () => Promise; getPushes: (query: Partial) => Promise; getPushesForUserProfile: (emailVariants: string[], profileUsername: string) => Promise; diff --git a/src/service/index.ts b/src/service/index.ts index 831a1c0fb..893ceaf1e 100644 --- a/src/service/index.ts +++ b/src/service/index.ts @@ -134,6 +134,12 @@ const corsOptions: cors.CorsOptions = { * @param {Proxy} proxy A reference to the proxy, used to restart it when necessary. * @return {Promise} the express application */ +// Backend sink types that promise a persistent session store. If one of these +// is active and getSessionStore() returns undefined, express-session would +// silently fall back to MemoryStore — which loses sessions on restart and is +// unsafe in any multi-process deployment. Throw loudly instead. +const PERSISTENT_SESSION_BACKENDS = new Set(['mongo', 'postgres']); + async function createApp(proxy: Proxy): Promise { // configuration of passport is async // Before we can bind the routes - we need the passport strategy @@ -143,9 +149,20 @@ async function createApp(proxy: Proxy): Promise { app.set('trust proxy', 1); app.use(limiter); + const backendType = config.getDatabase().type; + if (PERSISTENT_SESSION_BACKENDS.has(backendType)) { + await db.ensureSessionStoreReady(); + } + const sessionStore = db.getSessionStore(); + if (PERSISTENT_SESSION_BACKENDS.has(backendType) && !sessionStore) { + throw new Error( + `Session store for backend "${backendType}" failed to initialize — refusing to fall back to MemoryStore`, + ); + } + app.use( session({ - store: db.getSessionStore(), + store: sessionStore, secret: config.getCookieSecret(), resave: false, saveUninitialized: false, diff --git a/test-integration.postgres.proxy.config.json b/test-integration.postgres.proxy.config.json new file mode 100644 index 000000000..3885d004f --- /dev/null +++ b/test-integration.postgres.proxy.config.json @@ -0,0 +1,20 @@ +{ + "cookieSecret": "integration-test-cookie-secret", + "sessionMaxAgeHours": 12, + "sink": [ + { + "type": "fs", + "enabled": false + }, + { + "type": "postgres", + "enabled": true + } + ], + "authentication": [ + { + "type": "local", + "enabled": true + } + ] +} diff --git a/test/db/postgres/helper.test.ts b/test/db/postgres/helper.test.ts new file mode 100644 index 000000000..78336f579 --- /dev/null +++ b/test/db/postgres/helper.test.ts @@ -0,0 +1,758 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +const mockPoolQuery = vi.fn(); +const mockPoolEnd = vi.fn(); +const mockPoolCtor = vi.fn(); +const mockPoolOn = vi.fn(); +const mockPoolConnect = vi.fn(); +const mockClientQuery = vi.fn(); +const mockClientRelease = vi.fn(); + +vi.mock('pg', () => { + class Pool { + constructor(opts: unknown) { + mockPoolCtor(opts); + } + query = mockPoolQuery; + end = mockPoolEnd; + on = mockPoolOn; + connect = mockPoolConnect; + } + return { Pool }; +}); + +// connect-pg-simple returns a constructor that accepts options including a +// `pool` instance. We don't exercise the real store — just want to capture the +// options the helper passes. +const mockStoreCtor = vi.fn(); +vi.mock('connect-pg-simple', () => ({ + default: () => + class FakePgStore { + constructor(opts: unknown) { + mockStoreCtor(opts); + } + get(_sid: string, cb: (err: Error | null) => void) { + mockPoolQuery('SELECT 1', []); + cb(null); + } + close() { + return Promise.resolve(); + } + }, +})); + +const getDatabaseMock = vi.fn(); +vi.mock('../../../src/config', () => ({ + getDatabase: getDatabaseMock, +})); + +// Stand in for the optional @aws-sdk/rds-signer dependency so the IAM token +// path can be exercised without real AWS credentials. +const mockGetAuthToken = vi.fn(); +const mockSignerCtor = vi.fn(); +vi.mock('@aws-sdk/rds-signer', () => ({ + Signer: class { + constructor(opts: unknown) { + mockSignerCtor(opts); + } + getAuthToken = mockGetAuthToken; + }, +})); + +describe('PostgreSQL - helper', async () => { + const { + connect, + query, + resetConnection, + getSessionStore, + ensureSessionStoreReady, + withTransaction, + } = await import('../../../src/db/postgres/helper'); + + beforeEach(async () => { + vi.clearAllMocks(); + await resetConnection(); + mockPoolQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + mockGetAuthToken.mockResolvedValue('iam-token-123'); + mockClientQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + mockPoolConnect.mockResolvedValue({ query: mockClientQuery, release: mockClientRelease }); + }); + + describe('connect / migrations', () => { + it('runs migrations exactly once across many concurrent connects', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + await Promise.all([connect(), connect(), connect()]); + + // Pool constructed once; a single client acquired to run migrations once. + expect(mockPoolCtor).toHaveBeenCalledTimes(1); + expect(mockPoolConnect).toHaveBeenCalledTimes(1); + + const sqls = mockClientQuery.mock.calls.map((call) => String(call[0])); + expect(sqls[0]).toBe('BEGIN'); + expect(sqls.some((sql) => /pg_advisory_xact_lock/.test(sql))).toBe(true); + expect(sqls.some((sql) => /CREATE TABLE IF NOT EXISTS schema_migrations/.test(sql))).toBe( + true, + ); + expect(sqls.some((sql) => /CREATE TABLE IF NOT EXISTS users/.test(sql))).toBe(true); + expect(sqls[sqls.length - 1]).toBe('COMMIT'); + }); + + it('verifies instead of migrating when autoMigrate is false', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + autoMigrate: false, + }); + + const { MIGRATIONS } = await import('../../../src/db/postgres/schemaMigrations'); + mockPoolQuery + .mockResolvedValueOnce({ rowCount: 1, rows: [{ table_oid: 'schema_migrations' }] }) + .mockResolvedValueOnce({ + rowCount: MIGRATIONS.length, + rows: MIGRATIONS.map((m) => ({ version: m.version })), + }); + + await connect(); + + // No migration client acquired: nothing ran any DDL. + expect(mockPoolConnect).not.toHaveBeenCalled(); + const sqls = mockPoolQuery.mock.calls.map((call) => String(call[0])); + expect(sqls.some((sql) => /to_regclass/.test(sql))).toBe(true); + }); + + it('refuses to start when autoMigrate is false and migrations are pending', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + autoMigrate: false, + }); + + // A brand-new database: the bookkeeping table does not even exist. + mockPoolQuery.mockResolvedValueOnce({ rowCount: 1, rows: [{ table_oid: null }] }); + + await expect(connect()).rejects.toThrow(/pending migrations: 1 \(initial_schema\)/); + expect(mockPoolConnect).not.toHaveBeenCalled(); + }); + + it('retries migrations on the next call if they failed', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + // First migration run rejects on its opening statement. + mockClientQuery.mockRejectedValueOnce(new Error('schema kaboom')); + + await expect(connect()).rejects.toThrow('schema kaboom'); + + // The latch is cleared on failure, so the next connect re-runs migrations + // rather than being permanently latched to the rejected promise. + await connect(); + expect(mockPoolConnect).toHaveBeenCalledTimes(2); + }); + + it('throws when no connection is configured', async () => { + const PG_VARS = ['PGHOST', 'PGHOSTADDR', 'PGUSER', 'PGDATABASE'] as const; + const saved = PG_VARS.map((v) => [v, process.env[v]] as const); + for (const v of PG_VARS) delete process.env[v]; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: undefined, + }); + + await expect(query('SELECT 1')).rejects.toThrow('Postgres connection is not configured'); + + for (const [v, val] of saved) { + if (val !== undefined) process.env[v] = val; + } + }); + + it('accepts a PG* env setup that has no PGHOST (for example PGUSER/PGDATABASE)', async () => { + const PG_VARS = ['PGHOST', 'PGHOSTADDR', 'PGUSER', 'PGDATABASE'] as const; + const saved = PG_VARS.map((v) => [v, process.env[v]] as const); + for (const v of PG_VARS) delete process.env[v]; + process.env.PGUSER = 'gitproxy'; + process.env.PGDATABASE = 'gitproxy'; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: undefined, + }); + + await expect(query('SELECT 1')).resolves.toBeDefined(); + + for (const [v, val] of saved) { + if (val === undefined) delete process.env[v]; + else process.env[v] = val; + } + }); + }); + + describe('connection config', () => { + it('warns when a connection string overrides discrete fields', async () => { + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + host: 'ignored-host', + }); + + await query('SELECT 1'); + + expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('ignoring the discrete')); + warnSpy.mockRestore(); + }); + + it('does not warn when only a connection string is set', async () => { + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + await query('SELECT 1'); + + expect(warnSpy).not.toHaveBeenCalled(); + warnSpy.mockRestore(); + }); + + it('uses the connection string when provided', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({ connectionString: 'postgresql://localhost/x' }); + }); + + it('builds the pool from discrete fields when no connection string is set', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'db.example.com', + port: 5433, + user: 'gp', + password: 'secret', + database: 'gitproxy', + }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({ + host: 'db.example.com', + port: 5433, + user: 'gp', + password: 'secret', + database: 'gitproxy', + }); + }); + + it('prefers the connection string over discrete fields', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + host: 'ignored', + }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({ connectionString: 'postgresql://localhost/x' }); + }); + + it('falls through to PG* env vars when the sink has no explicit connection', async () => { + const savedPgHost = process.env.PGHOST; + process.env.PGHOST = 'env-host'; + getDatabaseMock.mockReturnValue({ type: 'postgres', enabled: true }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({}); + if (savedPgHost === undefined) delete process.env.PGHOST; + else process.env.PGHOST = savedPgHost; + }); + }); + + describe('ssl / TLS options', () => { + it('applies ssl=true alongside a connection string', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + ssl: true, + }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({ + connectionString: 'postgresql://localhost/x', + ssl: true, + }); + }); + + it('passes an ssl options object through to the pool', async () => { + const ssl = { rejectUnauthorized: false, ca: 'CA_CERT' }; + getDatabaseMock.mockReturnValue({ type: 'postgres', enabled: true, host: 'db', ssl }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({ host: 'db', ssl }); + }); + }); + + describe('pool tuning', () => { + it('applies pool options on top of the connection', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + pool: { max: 20, idleTimeoutMillis: 1000, connectionTimeoutMillis: 2000 }, + }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({ + connectionString: 'postgresql://localhost/x', + max: 20, + idleTimeoutMillis: 1000, + connectionTimeoutMillis: 2000, + }); + }); + + it('only sets the pool options that are provided', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'db', + pool: { max: 5 }, + }); + await connect(); + expect(mockPoolCtor).toHaveBeenCalledWith({ host: 'db', max: 5 }); + }); + }); + + describe('AWS RDS IAM authentication', () => { + const getOpts = () => mockPoolCtor.mock.calls[0][0] as Record; + + it('uses a generated IAM token as the password and defaults TLS on', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + port: 5432, + user: 'gp', + database: 'gitproxy', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + + const opts = getOpts(); + expect(opts.host).toBe('rds.example.com'); + expect(opts.port).toBe(5432); + expect(opts.user).toBe('gp'); + expect(opts.database).toBe('gitproxy'); + // RDS IAM mandates TLS, so it defaults on when ssl is not configured. + expect(opts.ssl).toBe(true); + // No static password — a token provider function instead. + expect(opts.connectionString).toBeUndefined(); + expect(typeof opts.password).toBe('function'); + + const token = await opts.password(); + expect(token).toBe('iam-token-123'); + expect(mockSignerCtor).toHaveBeenCalledWith({ + hostname: 'rds.example.com', + port: 5432, + username: 'gp', + region: 'eu-west-2', + }); + }); + + it('respects an explicit ssl setting instead of forcing true', async () => { + const ssl = { rejectUnauthorized: true, ca: 'RDS_CA' }; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + ssl, + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + expect(getOpts().ssl).toEqual(ssl); + }); + + it('warns that the connection string is ignored when IAM auth is enabled', async () => { + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://ignored/x', + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + + expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('ignoring connectionString')); + warnSpy.mockRestore(); + }); + + it('warns when ssl defaults to true in IAM mode (RDS CA is not in the default trust store)', async () => { + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + + expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('ssl.ca')); + warnSpy.mockRestore(); + }); + + it('does not warn about ssl when a CA bundle is supplied', async () => { + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + ssl: { rejectUnauthorized: true, ca: 'RDS_CA' }, + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + + const sslWarnings = warnSpy.mock.calls.filter(([m]) => String(m).includes('ssl.ca')); + expect(sslWarnings).toEqual([]); + warnSpy.mockRestore(); + }); + + it('ignores a connection string when IAM auth is enabled', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://ignored/x', + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + const opts = getOpts(); + expect(opts.connectionString).toBeUndefined(); + expect(opts.host).toBe('rds.example.com'); + expect(typeof opts.password).toBe('function'); + }); + + it('falls back to AWS_REGION when no region is configured', async () => { + const savedRegion = process.env.AWS_REGION; + process.env.AWS_REGION = 'us-east-1'; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true }, + }); + + await connect(); + await getOpts().password(); + expect(mockSignerCtor).toHaveBeenCalledWith(expect.objectContaining({ region: 'us-east-1' })); + + if (savedRegion === undefined) delete process.env.AWS_REGION; + else process.env.AWS_REGION = savedRegion; + }); + + it('throws a clear error when host or user cannot be resolved', async () => { + const savedPgUser = process.env.PGUSER; + delete process.env.PGUSER; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await expect(connect()).rejects.toThrow( + /AWS RDS IAM authentication requires `host` and `user`/, + ); + + if (savedPgUser !== undefined) process.env.PGUSER = savedPgUser; + }); + + it('propagates a token-generation failure to the connection', async () => { + mockGetAuthToken.mockRejectedValueOnce(new Error('STS denied')); + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + await expect(getOpts().password()).rejects.toThrow('STS denied'); + }); + + it('falls back to AWS_DEFAULT_REGION when AWS_REGION is unset', async () => { + const savedRegion = process.env.AWS_REGION; + const savedDefault = process.env.AWS_DEFAULT_REGION; + delete process.env.AWS_REGION; + process.env.AWS_DEFAULT_REGION = 'ap-south-1'; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true }, + }); + + await connect(); + await getOpts().password(); + expect(mockSignerCtor).toHaveBeenCalledWith( + expect.objectContaining({ region: 'ap-south-1' }), + ); + + if (savedRegion === undefined) delete process.env.AWS_REGION; + else process.env.AWS_REGION = savedRegion; + if (savedDefault === undefined) delete process.env.AWS_DEFAULT_REGION; + else process.env.AWS_DEFAULT_REGION = savedDefault; + }); + + it('prefers AWS_REGION over AWS_DEFAULT_REGION', async () => { + const savedRegion = process.env.AWS_REGION; + const savedDefault = process.env.AWS_DEFAULT_REGION; + process.env.AWS_REGION = 'us-west-2'; + process.env.AWS_DEFAULT_REGION = 'ap-south-1'; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true }, + }); + + await connect(); + await getOpts().password(); + expect(mockSignerCtor).toHaveBeenCalledWith(expect.objectContaining({ region: 'us-west-2' })); + + if (savedRegion === undefined) delete process.env.AWS_REGION; + else process.env.AWS_REGION = savedRegion; + if (savedDefault === undefined) delete process.env.AWS_DEFAULT_REGION; + else process.env.AWS_DEFAULT_REGION = savedDefault; + }); + + it('defaults the IAM token port to 5432 when none is configured', async () => { + const savedPgPort = process.env.PGPORT; + delete process.env.PGPORT; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await connect(); + await getOpts().password(); + expect(mockSignerCtor).toHaveBeenCalledWith(expect.objectContaining({ port: 5432 })); + + if (savedPgPort !== undefined) process.env.PGPORT = savedPgPort; + }); + + it('throws an actionable error when @aws-sdk/rds-signer is not installed', async () => { + // Re-import the helper against a registry where the optional dependency + // fails to resolve, to exercise loadRdsSigner's catch branch — the exact + // failure a user hits after `npm install --omit=optional`. + vi.resetModules(); + vi.doMock('@aws-sdk/rds-signer', () => { + throw new Error('Cannot find module'); + }); + + const fresh = await import('../../../src/db/postgres/helper'); + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + host: 'rds.example.com', + user: 'gp', + awsIamAuth: { enabled: true, region: 'eu-west-2' }, + }); + + await fresh.connect(); + const opts = mockPoolCtor.mock.calls[0][0] as Record; + await expect(opts.password()).rejects.toThrow( + /requires the optional `@aws-sdk\/rds-signer` dependency/, + ); + + await fresh.resetConnection(); + vi.doUnmock('@aws-sdk/rds-signer'); + vi.resetModules(); + }); + }); + + describe('withTransaction', () => { + it('wraps the callback in BEGIN/COMMIT and releases the client', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + // Warm the pool first so the schema bootstrap's own client activity + // does not blend into the assertions below. + await connect(); + mockClientQuery.mockClear(); + mockClientRelease.mockClear(); + + const result = await withTransaction(async (client) => { + await client.query('SELECT 1'); + return 'ok'; + }); + + expect(result).toBe('ok'); + expect(mockClientQuery.mock.calls.map(([sql]) => sql)).toEqual([ + 'BEGIN', + 'SELECT 1', + 'COMMIT', + ]); + expect(mockClientRelease).toHaveBeenCalledTimes(1); + }); + + it('rolls back and rethrows when the callback fails', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + await connect(); + mockClientQuery.mockClear(); + mockClientRelease.mockClear(); + + await expect( + withTransaction(async () => { + throw new Error('boom'); + }), + ).rejects.toThrow('boom'); + + expect(mockClientQuery.mock.calls.map(([sql]) => sql)).toEqual(['BEGIN', 'ROLLBACK']); + expect(mockClientRelease).toHaveBeenCalledTimes(1); + }); + }); + + describe('pool error handling', () => { + it('registers an idle-client error listener that logs without crashing', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined); + + await connect(); + + const errorRegistration = mockPoolOn.mock.calls.find((call) => call[0] === 'error'); + expect(errorRegistration).toBeDefined(); + + const handler = errorRegistration![1] as (err: Error) => void; + expect(() => handler(new Error('connection terminated unexpectedly'))).not.toThrow(); + expect(errorSpy).toHaveBeenCalled(); + + errorSpy.mockRestore(); + }); + }); + + describe('getSessionStore', () => { + it('throws when no connection is configured — no MemoryStore fallback', () => { + // GitHub-hosted runners preset PGUSER/PGPASSWORD for their bundled + // postgres tooling, and the connection guard honours the PG* family, so + // every variable it reads must be cleared here. + const PG_VARS = ['PGHOST', 'PGHOSTADDR', 'PGUSER', 'PGDATABASE'] as const; + const saved = PG_VARS.map((v) => [v, process.env[v]] as const); + for (const v of PG_VARS) delete process.env[v]; + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: undefined, + }); + + expect(() => getSessionStore()).toThrow( + /Postgres connection is required for session storage/, + ); + + for (const [v, val] of saved) { + if (val !== undefined) process.env[v] = val; + } + }); + + it('passes the shared pool to connect-pg-simple without its own DDL path', () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + getSessionStore(); + + expect(mockStoreCtor).toHaveBeenCalledTimes(1); + const opts = mockStoreCtor.mock.calls[0][0] as Record; + expect(opts.tableName).toBe('session'); + // The session table is owned by the versioned migration list; the store + // creating it would be a second, unversioned DDL path. + expect(opts.createTableIfMissing).toBe(false); + expect(opts.pool).toBeDefined(); + }); + + it('touches the session store during readiness checks', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + await ensureSessionStoreReady(); + + expect(mockStoreCtor).toHaveBeenCalledTimes(1); + expect(mockPoolQuery).toHaveBeenCalled(); + }); + + it('runs migrations before probing the store, which no longer owns its DDL', async () => { + getDatabaseMock.mockReturnValue({ + type: 'postgres', + enabled: true, + connectionString: 'postgresql://localhost/x', + }); + + await ensureSessionStoreReady(); + + // The migration client ran (the session table is created by migration + // 7, so readiness must not probe an unmigrated database). + expect(mockPoolConnect).toHaveBeenCalledTimes(1); + const sqls = mockClientQuery.mock.calls.map((call) => String(call[0])); + expect(sqls.some((sql) => /CREATE TABLE IF NOT EXISTS "session"/.test(sql))).toBe(true); + }); + }); +}); diff --git a/test/db/postgres/migrate.integration.test.ts b/test/db/postgres/migrate.integration.test.ts new file mode 100644 index 000000000..c9d534215 --- /dev/null +++ b/test/db/postgres/migrate.integration.test.ts @@ -0,0 +1,96 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import fs from 'fs'; +import os from 'os'; +import path from 'path'; + +import Datastore from '@seald-io/nedb'; +import { describe, it, expect, afterAll } from 'vitest'; + +import * as postgres from '../../../src/db/postgres'; +import { migrate } from '../../../src/db/postgres/migrate'; +import { createFileSource } from '../../../src/db/postgres/migrateFileSource'; + +const shouldRunPostgresTests = process.env.RUN_POSTGRES_TESTS === 'true'; + +const tmpDirs: string[] = []; + +const seedFsBackend = async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gp-migrate-')); + tmpDirs.push(dir); + await new Datastore({ filename: path.join(dir, 'users.db'), autoload: true }).insertAsync({ + username: 'mig-alice', + email: 'mig-alice@x.com', + password: 'hash', + gitAccount: 'mig-alice-git', + admin: false, + }); + await new Datastore({ filename: path.join(dir, 'repos.db'), autoload: true }).insertAsync({ + project: 'mig', + name: 'mig-repo', + url: 'https://example.com/mig/repo.git', + users: { canPush: ['mig-alice'], canAuthorise: [] }, + }); + await new Datastore({ filename: path.join(dir, 'pushes.db'), autoload: true }).insertAsync({ + id: 'mig-push-1', + type: 'push', + timestamp: 1700000000000, + }); + return dir; +}; + +afterAll(() => { + for (const dir of tmpDirs) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe.runIf(shouldRunPostgresTests)('PostgreSQL Data Migration Integration Tests', () => { + it('migrates users, repos and pushes from an fs backend into postgres', async () => { + const source = createFileSource(await seedFsBackend()); + + const summary = await migrate(source, postgres); + await source.close(); + + expect(summary).toEqual({ + users: { imported: 1, skipped: 0 }, + repos: { imported: 1, skipped: 0 }, + pushes: { imported: 1 }, + }); + + const user = await postgres.findUser('mig-alice'); + expect(user?.email).toBe('mig-alice@x.com'); + expect(user?._id).toMatch(/^[0-9a-f-]{36}$/i); // freshly assigned UUID + + const repo = await postgres.getRepoByUrl('https://example.com/mig/repo.git'); + expect(repo?.users.canPush).toEqual(['mig-alice']); + + const push = await postgres.getPush('mig-push-1'); + expect(push?.id).toBe('mig-push-1'); + }); + + it('skips already-imported users and repos on a second run', async () => { + const source = createFileSource(await seedFsBackend()); + + await migrate(source, postgres); + const second = await migrate(source, postgres); + await source.close(); + + expect(second.users).toEqual({ imported: 0, skipped: 1 }); + expect(second.repos).toEqual({ imported: 0, skipped: 1 }); + // Pushes are upserted by id, so re-running still "imports" (writes) them. + expect(second.pushes.imported).toBe(1); + }); +}); diff --git a/test/db/postgres/migrate.test.ts b/test/db/postgres/migrate.test.ts new file mode 100644 index 000000000..89e61845b --- /dev/null +++ b/test/db/postgres/migrate.test.ts @@ -0,0 +1,194 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, vi } from 'vitest'; + +import { migrate, MigrationDestination, MigrationSource } from '../../../src/db/postgres/migrate'; +import { Repo, User } from '../../../src/db/types'; + +const user = (username: string, email: string): User => + new User(username, 'hash', `${username}-git`, email, false); + +const repo = (url: string): Repo => new Repo('proj', `name-${url}`, url); + +const makeSource = (over: Partial = {}): MigrationSource => ({ + getUsers: vi.fn().mockResolvedValue([]), + getRepos: vi.fn().mockResolvedValue([]), + getPushBatches: vi.fn(async function* () {}), + close: vi.fn().mockResolvedValue(undefined), + ...over, +}); + +const makeDestination = (over: Partial = {}): MigrationDestination => ({ + findUser: vi.fn().mockResolvedValue(null), + findUserByEmail: vi.fn().mockResolvedValue(null), + createUser: vi.fn().mockResolvedValue(undefined), + getRepoByUrl: vi.fn().mockResolvedValue(null), + createRepo: vi.fn().mockResolvedValue(undefined), + writeAudit: vi.fn().mockResolvedValue(undefined), + ...over, +}); + +describe('PostgreSQL - migrate', () => { + it('imports users, repos and pushes into an empty destination', async () => { + const source = makeSource({ + getUsers: vi.fn().mockResolvedValue([user('alice', 'alice@x.com'), user('bob', 'bob@x.com')]), + getRepos: vi.fn().mockResolvedValue([repo('https://x/a.git')]), + getPushBatches: vi.fn(async function* () { + yield [{ id: 'p1' }, { id: 'p2' }, { id: 'p3' }] as never; + }), + }); + const destination = makeDestination(); + + const summary = await migrate(source, destination as never); + + expect(summary).toEqual({ + users: { imported: 2, skipped: 0 }, + repos: { imported: 1, skipped: 0 }, + pushes: { imported: 3 }, + }); + expect(destination.createUser).toHaveBeenCalledTimes(2); + expect(destination.createRepo).toHaveBeenCalledTimes(1); + expect(destination.writeAudit).toHaveBeenCalledTimes(3); + }); + + it('defaults missing email and gitAccount on legacy users', async () => { + const legacy = user('ad-user', 'ignored'); + delete (legacy as Partial).email; + delete (legacy as Partial).gitAccount; + const source = makeSource({ getUsers: vi.fn().mockResolvedValue([legacy]) }); + const destination = makeDestination(); + + const summary = await migrate(source, destination as never); + + expect(summary.users).toEqual({ imported: 1, skipped: 0 }); + expect(destination.createUser).toHaveBeenCalledWith( + expect.objectContaining({ username: 'ad-user', email: '', gitAccount: '' }), + ); + // No email to dedupe on, so the email lookup is skipped entirely. + expect(destination.findUserByEmail).not.toHaveBeenCalled(); + }); + + it('skips a user that already exists by username', async () => { + const source = makeSource({ getUsers: vi.fn().mockResolvedValue([user('alice', 'a@x.com')]) }); + const destination = makeDestination({ + findUser: vi.fn().mockResolvedValue(user('alice', 'a@x.com')), + }); + + const summary = await migrate(source, destination as never); + + expect(summary.users).toEqual({ imported: 0, skipped: 1 }); + expect(destination.createUser).not.toHaveBeenCalled(); + }); + + it('skips a user that already exists by email when the username differs', async () => { + const source = makeSource({ getUsers: vi.fn().mockResolvedValue([user('alice', 'a@x.com')]) }); + const destination = makeDestination({ + findUser: vi.fn().mockResolvedValue(null), + findUserByEmail: vi.fn().mockResolvedValue(user('other', 'a@x.com')), + }); + + const summary = await migrate(source, destination as never); + + expect(summary.users).toEqual({ imported: 0, skipped: 1 }); + expect(destination.createUser).not.toHaveBeenCalled(); + }); + + it('skips a repo that already exists by URL', async () => { + const source = makeSource({ getRepos: vi.fn().mockResolvedValue([repo('https://x/a.git')]) }); + const destination = makeDestination({ + getRepoByUrl: vi.fn().mockResolvedValue(repo('https://x/a.git')), + }); + + const summary = await migrate(source, destination as never); + + expect(summary.repos).toEqual({ imported: 0, skipped: 1 }); + expect(destination.createRepo).not.toHaveBeenCalled(); + }); + + it('does not look up by email when the source user has none', async () => { + const noEmail = new User('svc', 'hash', 'svc-git', '', false); + const source = makeSource({ getUsers: vi.fn().mockResolvedValue([noEmail]) }); + const destination = makeDestination(); + + await migrate(source, destination as never); + + expect(destination.findUserByEmail).not.toHaveBeenCalled(); + expect(destination.createUser).toHaveBeenCalledTimes(1); + }); + + it('reports progress through the supplied logger', async () => { + const source = makeSource({ getUsers: vi.fn().mockResolvedValue([user('a', 'a@x.com')]) }); + const log = vi.fn(); + + await migrate(source, makeDestination() as never, { log }); + + expect(log).toHaveBeenCalledWith('Migrating 1 user(s)...'); + expect(log).toHaveBeenCalledWith('Migrating 0 repo(s)...'); + expect(log).toHaveBeenCalledWith('Migrating pushes...'); + }); + + it('writes pushes batch by batch and counts across batches', async () => { + const source = makeSource({ + getPushBatches: vi.fn(async function* () { + yield [{ id: 'p1' }, { id: 'p2' }] as never; + yield [{ id: 'p3' }] as never; + }), + }); + const destination = makeDestination(); + + const summary = await migrate(source, destination as never); + + expect(summary.pushes).toEqual({ imported: 3 }); + expect(destination.writeAudit).toHaveBeenCalledTimes(3); + }); + + it('passes user passwords through unchanged (already hashed at the source)', async () => { + // The destination is the raw postgres adapter, whose createUser stores the + // supplied value verbatim; hashing lives in the service-level wrapper in + // src/db/index.ts. Source passwords are bcrypt hashes and must survive + // the copy byte for byte, or every migrated login would break. + const hashed = '$2a$10$abcdefghijklmnopqrstuv'; + const source = makeSource({ + getUsers: vi + .fn() + .mockResolvedValue([Object.assign(user('alice', 'alice@x.com'), { password: hashed })]), + }); + const destination = makeDestination(); + + await migrate(source, destination as never); + + expect(destination.createUser).toHaveBeenCalledWith( + expect.objectContaining({ password: hashed }), + ); + }); + + it('passes repo permissions through createRepo without separate role calls', async () => { + // The adapter's createRepo persists the whole users permission map, so the + // migration needs no addUserCanPush / addUserCanAuthorise follow-ups. + const seeded = Object.assign(repo('https://x/a.git'), { + users: { canPush: ['alice'], canAuthorise: ['bob'] }, + }); + const source = makeSource({ getRepos: vi.fn().mockResolvedValue([seeded]) }); + const destination = makeDestination(); + + await migrate(source, destination as never); + + expect(destination.createRepo).toHaveBeenCalledWith( + expect.objectContaining({ users: { canPush: ['alice'], canAuthorise: ['bob'] } }), + ); + }); +}); diff --git a/test/db/postgres/migrateFileSource.test.ts b/test/db/postgres/migrateFileSource.test.ts new file mode 100644 index 000000000..d681dea20 --- /dev/null +++ b/test/db/postgres/migrateFileSource.test.ts @@ -0,0 +1,101 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import fs from 'fs'; +import os from 'os'; +import path from 'path'; + +import Datastore from '@seald-io/nedb'; +import { describe, it, expect, afterAll } from 'vitest'; + +import { createFileSource } from '../../../src/db/postgres/migrateFileSource'; + +const tmpDirs: string[] = []; + +const makeDataDir = () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gp-fs-src-')); + tmpDirs.push(dir); + return dir; +}; + +const seed = (dir: string, file: string, doc: Record) => + new Datastore({ filename: path.join(dir, file), autoload: true }).insertAsync(doc); + +afterAll(() => { + for (const dir of tmpDirs) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('PostgreSQL - migrate file source', () => { + it('reads users, repos and pushes from the NeDB datastores', async () => { + const dir = makeDataDir(); + await seed(dir, 'users.db', { + username: 'alice', + email: 'alice@x.com', + gitAccount: 'a', + admin: true, + }); + await seed(dir, 'repos.db', { + project: 'p', + name: 'n', + url: 'https://x/n.git', + users: { canPush: [], canAuthorise: [] }, + }); + await seed(dir, 'pushes.db', { id: 'push-1', type: 'push' }); + + const source = createFileSource(dir); + + expect((await source.getUsers()).map((u) => u.username)).toEqual(['alice']); + expect((await source.getRepos()).map((r) => r.url)).toEqual(['https://x/n.git']); + const pushes: { id?: string }[] = []; + for await (const batch of source.getPushBatches(100)) pushes.push(...batch); + expect(pushes.map((p) => p.id)).toEqual(['push-1']); + + await source.close(); + }); + + it('returns empty results when the datastores exist but have no records', async () => { + const dir = makeDataDir(); + for (const file of ['users.db', 'repos.db', 'pushes.db']) { + fs.writeFileSync(path.join(dir, file), ''); + } + const source = createFileSource(dir); + + expect(await source.getUsers()).toEqual([]); + expect(await source.getRepos()).toEqual([]); + const batches: unknown[] = []; + for await (const batch of source.getPushBatches(100)) batches.push(batch); + expect(batches).toEqual([]); + }); + + it('fails fast when the data directory does not exist', () => { + expect(() => createFileSource(path.join(os.tmpdir(), 'gp-definitely-missing'))).toThrow( + /does not exist/, + ); + }); + + it('fails fast when the directory holds no fs sink datastores', () => { + expect(() => createFileSource(makeDataDir())).toThrow(/No fs sink datastores/); + }); + + it('reports a corrupt datastore instead of treating it as empty', async () => { + const dir = makeDataDir(); + fs.writeFileSync(path.join(dir, 'users.db'), 'this is not nedb json\n{broken'); + + const source = createFileSource(dir); + + await expect(source.getUsers()).rejects.toThrow(/Failed to load/); + }); +}); diff --git a/test/db/postgres/migrateMongoSource.test.ts b/test/db/postgres/migrateMongoSource.test.ts new file mode 100644 index 000000000..1806065f5 --- /dev/null +++ b/test/db/postgres/migrateMongoSource.test.ts @@ -0,0 +1,99 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +const mockConnect = vi.fn(); +const mockClose = vi.fn(); +const mockCtor = vi.fn(); + +const docsByCollection: Record = {}; + +vi.mock('mongodb', () => ({ + MongoClient: class { + constructor(connectionString: string, options: unknown) { + mockCtor(connectionString, options); + } + connect = mockConnect; + close = mockClose; + db() { + return { + collection: (name: string) => ({ + find: () => ({ + toArray: () => Promise.resolve(docsByCollection[name] ?? []), + batchSize: (_n: number) => ({ + async *[Symbol.asyncIterator]() { + for (const doc of docsByCollection[name] ?? []) yield doc; + }, + }), + }), + }), + }; + } + }, +})); + +describe('PostgreSQL - migrate mongo source', async () => { + const { createMongoSource } = await import('../../../src/db/postgres/migrateMongoSource'); + + beforeEach(() => { + vi.clearAllMocks(); + docsByCollection.users = [ + { _id: 'objid-1', username: 'alice', email: 'alice@x.com', gitAccount: 'a', admin: true }, + ]; + docsByCollection.repos = [{ _id: 'objid-2', project: 'p', name: 'n', url: 'https://x/n.git' }]; + docsByCollection.pushes = [{ _id: 'objid-3', id: 'push-1', type: 'push' }]; + }); + + it('connects with the supplied connection string and options', async () => { + await createMongoSource('mongodb://localhost/src', { tls: true }); + expect(mockCtor).toHaveBeenCalledWith('mongodb://localhost/src', { tls: true }); + expect(mockConnect).toHaveBeenCalledTimes(1); + }); + + it('reads users, repos and pushes from their collections', async () => { + const source = await createMongoSource('mongodb://localhost/src'); + + const users = await source.getUsers(); + const repos = await source.getRepos(); + const pushes: { id?: string }[] = []; + for await (const batch of source.getPushBatches(100)) pushes.push(...batch); + + expect(users.map((u) => u.username)).toEqual(['alice']); + expect(repos.map((r) => r.url)).toEqual(['https://x/n.git']); + expect(pushes.map((p) => p.id)).toEqual(['push-1']); + }); + + it('closes the underlying client', async () => { + const source = await createMongoSource('mongodb://localhost/src'); + await source.close(); + expect(mockClose).toHaveBeenCalledTimes(1); + }); + + it('splits pushes into batches of the requested size', async () => { + docsByCollection.pushes = [ + { id: 'push-1', type: 'push' }, + { id: 'push-2', type: 'push' }, + { id: 'push-3', type: 'push' }, + ]; + const source = await createMongoSource('mongodb://localhost/src'); + + const batches: unknown[][] = []; + for await (const batch of source.getPushBatches(2)) batches.push(batch); + + expect(batches.map((b) => b.length)).toEqual([2, 1]); + }); +}); diff --git a/test/db/postgres/migrations.test.ts b/test/db/postgres/migrations.test.ts new file mode 100644 index 000000000..7d40a952a --- /dev/null +++ b/test/db/postgres/migrations.test.ts @@ -0,0 +1,78 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +const mockQuery = vi.fn(); + +vi.mock('../../../src/db/postgres/helper', () => ({ + query: mockQuery, +})); + +describe('PostgreSQL - Migrations', async () => { + const { deriveCreatedAt, getAppliedMigrations, recordMigration, unrecordMigration } = + await import('../../../src/db/postgres/migrations'); + + beforeEach(() => { + vi.clearAllMocks(); + }); + + describe('deriveCreatedAt', () => { + it('cannot recover a timestamp from a random UUID', () => { + // Same contract as the filesystem backend: callers fall back to their own default. + expect(deriveCreatedAt()).toBeUndefined(); + }); + }); + + describe('getAppliedMigrations', () => { + it('returns the recorded ids', async () => { + mockQuery.mockResolvedValue({ rowCount: 2, rows: [{ id: '001-a' }, { id: '002-b' }] }); + + await expect(getAppliedMigrations()).resolves.toEqual(['001-a', '002-b']); + }); + + it('returns an empty list on a fresh database', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await expect(getAppliedMigrations()).resolves.toEqual([]); + }); + }); + + describe('recordMigration', () => { + it('is idempotent so an interrupted run can resume', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await recordMigration('001-a'); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('INSERT INTO migrations'); + expect(sql).toContain('ON CONFLICT (id) DO NOTHING'); + expect(params).toEqual(['001-a']); + }); + }); + + describe('unrecordMigration', () => { + it('deletes only the given id', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await unrecordMigration('001-a'); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('DELETE FROM migrations WHERE id = $1'); + expect(params).toEqual(['001-a']); + }); + }); +}); diff --git a/test/db/postgres/pushes.integration.test.ts b/test/db/postgres/pushes.integration.test.ts new file mode 100644 index 000000000..6e3c8b779 --- /dev/null +++ b/test/db/postgres/pushes.integration.test.ts @@ -0,0 +1,298 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, beforeEach } from 'vitest'; +import { + writeAudit, + getPush, + getPushes, + deletePush, + authorise, + reject, + cancel, +} from '../../../src/db/postgres/pushes'; +import { Action } from '../../../src/proxy/actions'; + +const shouldRunPostgresTests = process.env.RUN_POSTGRES_TESTS === 'true'; + +describe.runIf(shouldRunPostgresTests)('PostgreSQL Pushes Integration Tests', () => { + const createTestAction = (overrides: Partial = {}): Action => { + const timestamp = Date.now(); + const action = new Action( + overrides.id || `test-push-${timestamp}`, + overrides.type || 'push', + overrides.method || 'POST', + overrides.timestamp || timestamp, + overrides.url || 'https://github.com/test/repo.git', + ); + + action.error = overrides.error ?? false; + action.blocked = overrides.blocked ?? true; + action.allowPush = overrides.allowPush ?? false; + action.authorised = overrides.authorised ?? false; + action.canceled = overrides.canceled ?? false; + action.rejected = overrides.rejected ?? false; + + return action; + }; + + describe('writeAudit', () => { + it('writes an action to the database', async () => { + const action = createTestAction({ id: 'write-audit-test' }); + await writeAudit(action); + + const retrieved = await getPush('write-audit-test'); + expect(retrieved).not.toBeNull(); + expect(retrieved?.id).toBe('write-audit-test'); + }); + + it('upserts an existing action', async () => { + const action = createTestAction({ id: 'upsert-test' }); + await writeAudit(action); + + action.blocked = false; + action.allowPush = true; + await writeAudit(action); + + const retrieved = await getPush('upsert-test'); + expect(retrieved?.blocked).toBe(false); + expect(retrieved?.allowPush).toBe(true); + }); + + it('throws Invalid id for non-string ids', async () => { + const action = createTestAction(); + action.id = 123 as unknown as string; + + await expect(writeAudit(action)).rejects.toThrow('Invalid id'); + }); + + it('strips _id from action before saving', async () => { + const action = createTestAction({ id: 'strip-id-test' }); + (action as any)._id = 'should-be-removed'; + + await writeAudit(action); + const retrieved = await getPush('strip-id-test'); + expect(retrieved).not.toBeNull(); + // _id should not leak back out — the action JSON contains only public fields + expect((retrieved as any)._id).toBeUndefined(); + expect(retrieved?.id).toBe('strip-id-test'); + }); + }); + + describe('getPush', () => { + it('retrieves a push by id', async () => { + const action = createTestAction({ id: 'get-push-test' }); + await writeAudit(action); + + const result = await getPush('get-push-test'); + expect(result?.id).toBe('get-push-test'); + expect(result?.type).toBe('push'); + }); + + it('returns null for a non-existent push', async () => { + expect(await getPush('non-existent')).toBeNull(); + }); + + it('returns an Action instance', async () => { + const action = createTestAction({ id: 'action-instance-test' }); + await writeAudit(action); + + const result = await getPush('action-instance-test'); + expect(Object.getPrototypeOf(result)).toBe(Action.prototype); + }); + }); + + describe('getPushes', () => { + beforeEach(async () => { + // Three pushes with deliberately increasing timestamps so we can verify + // DESC ordering deterministically. + await writeAudit( + createTestAction({ + id: 'push-a', + timestamp: 1000, + blocked: true, + authorised: false, + }), + ); + await writeAudit( + createTestAction({ + id: 'push-b', + timestamp: 2000, + blocked: true, + authorised: false, + }), + ); + await writeAudit( + createTestAction({ + id: 'push-authorised', + timestamp: 3000, + blocked: true, + authorised: true, + }), + ); + }); + + it('orders pushes by timestamp DESC', async () => { + const result = await getPushes({}); + const ids = result.map((p) => p.id); + expect(ids).toEqual(['push-authorised', 'push-b', 'push-a']); + }); + + it('filters by authorised flag', async () => { + const result = await getPushes({ authorised: true }); + const authorisedPush = result.find((p) => p.id === 'push-authorised'); + expect(authorisedPush).toBeDefined(); + expect(result.every((p) => p.authorised === true)).toBe(true); + }); + + it('does not leak _id', async () => { + const result = await getPushes({}); + result.forEach((push) => { + expect((push as any)._id).toBeUndefined(); + expect(push.id).toBeDefined(); + }); + }); + }); + + describe('deletePush', () => { + it('deletes a push by id', async () => { + const action = createTestAction({ id: 'delete-test' }); + await writeAudit(action); + await deletePush('delete-test'); + expect(await getPush('delete-test')).toBeNull(); + }); + + it('does not throw when deleting a non-existent push', async () => { + await expect(deletePush('non-existent')).resolves.not.toThrow(); + }); + }); + + describe('authorise', () => { + it('authorises a push and resets cancel/reject flags', async () => { + const action = createTestAction({ + id: 'authorise-test', + authorised: false, + canceled: true, + rejected: true, + }); + await writeAudit(action); + + const result = await authorise('authorise-test', { note: 'approved' } as never); + expect(result.message).toBe('authorised authorise-test'); + + const updated = await getPush('authorise-test'); + expect(updated?.authorised).toBe(true); + expect(updated?.canceled).toBe(false); + expect(updated?.rejected).toBe(false); + expect((updated as any)?.attestation).toEqual({ note: 'approved' }); + }); + + it('throws for a non-existent push', async () => { + await expect(authorise('non-existent', {} as never)).rejects.toThrow( + 'push non-existent not found', + ); + }); + }); + + describe('reject', () => { + it('rejects a push and persists the rejection payload', async () => { + const action = createTestAction({ + id: 'reject-test', + authorised: true, + canceled: true, + rejected: false, + }); + await writeAudit(action); + + const rejection = { + reason: 'policy violation', + timestamp: new Date('2026-05-11T00:00:00Z'), + reviewer: { username: 'r', reviewerEmail: 'r@example.com' }, + }; + + const result = await reject('reject-test', rejection as never); + expect(result.message).toBe('reject reject-test'); + + const updated = await getPush('reject-test'); + expect(updated?.authorised).toBe(false); + expect(updated?.canceled).toBe(false); + expect(updated?.rejected).toBe(true); + // Round-tripped through JSONB — `reason` and `reviewer` survive + // exactly; the `Date` round-trips as an ISO string in JSON. + expect((updated as any)?.rejection?.reason).toBe('policy violation'); + expect((updated as any)?.rejection?.reviewer).toEqual(rejection.reviewer); + }); + + it('throws for a non-existent push', async () => { + await expect(reject('non-existent', {} as never)).rejects.toThrow( + 'push non-existent not found', + ); + }); + }); + + describe('cancel', () => { + it('cancels a push and resets authorise/reject flags', async () => { + const action = createTestAction({ + id: 'cancel-test', + authorised: true, + canceled: false, + rejected: true, + }); + await writeAudit(action); + + const result = await cancel('cancel-test'); + expect(result.message).toBe('canceled cancel-test'); + + const updated = await getPush('cancel-test'); + expect(updated?.authorised).toBe(false); + expect(updated?.canceled).toBe(true); + expect(updated?.rejected).toBe(false); + }); + + it('throws for a non-existent push', async () => { + await expect(cancel('non-existent')).rejects.toThrow('push non-existent not found'); + }); + }); + + describe('concurrent decisions', () => { + it('serialises concurrent authorise/reject so the final state is one coherent decision', async () => { + // Each decision reads the row FOR UPDATE inside a transaction, so the + // two calls run one after the other; whichever commits last defines the + // final state, but the flags can never blend into an inconsistent mix + // (e.g. authorised AND rejected both true) and neither write is lost + // mid-read-modify-write. + const action = createTestAction({ id: 'decision-race-test' }); + await writeAudit(action); + + const rejection = { + reason: { message: 'concurrent reject' }, + timestamp: new Date(), + reviewer: { username: 'reviewer', reviewerEmail: 'reviewer@example.com' }, + }; + await Promise.all([ + authorise('decision-race-test'), + reject('decision-race-test', rejection as never), + ]); + + const final = await getPush('decision-race-test'); + const flags = [final?.authorised, final?.rejected].filter(Boolean); + expect(flags).toHaveLength(1); + if (final?.rejected) { + expect(final.rejection).toBeTruthy(); + } + }); + }); +}); diff --git a/test/db/postgres/pushes.test.ts b/test/db/postgres/pushes.test.ts new file mode 100644 index 000000000..5a91a5eab --- /dev/null +++ b/test/db/postgres/pushes.test.ts @@ -0,0 +1,356 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +const mockQuery = vi.fn(); + +vi.mock('../../../src/db/postgres/helper', () => ({ + query: mockQuery, + // Runs the callback with a client whose query records into the same mock, + // so tests assert the statement sequence; transactional semantics themselves + // are covered by the withTransaction tests in helper.test.ts. + withTransaction: (fn: (client: { query: typeof mockQuery }) => Promise) => + fn({ query: mockQuery }), +})); + +describe('PostgreSQL - Pushes', async () => { + const { + reject, + getPushes, + getPush, + writeAudit, + authorise, + cancel, + deletePush, + getPushesForUserProfile, + getRepoPushRollupsByCanonicalUrl, + } = await import('../../../src/db/postgres/pushes'); + + beforeEach(() => { + vi.clearAllMocks(); + }); + + describe('getPushes', () => { + it('orders results by timestamp DESC', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await getPushes({}); + + const [sql] = mockQuery.mock.calls[0]; + expect(sql).toMatch(/ORDER BY timestamp DESC/); + }); + + it('translates allowPush to the snake_case column', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await getPushes({ allowPush: true }); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('allow_push = $1'); + expect(params).toEqual([true]); + }); + + it('ignores unknown filter keys', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await getPushes({ id: 'x' } as never); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).not.toContain('WHERE'); + expect(params).toEqual([]); + }); + }); + + describe('getPush', () => { + it('returns null when no row matches', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + expect(await getPush('missing')).toBeNull(); + }); + }); + + describe('writeAudit', () => { + it('throws Invalid id when id is not a string', async () => { + const action = { id: 42, timestamp: 1 } as unknown as Parameters[0]; + await expect(writeAudit(action)).rejects.toThrow('Invalid id'); + expect(mockQuery).not.toHaveBeenCalled(); + }); + + it('upserts via ON CONFLICT (id)', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + const action = { + id: 'push-1', + timestamp: 1234, + type: 'push', + error: false, + blocked: true, + allowPush: false, + authorised: false, + canceled: false, + rejected: false, + } as unknown as Parameters[0]; + + await writeAudit(action); + + const [sql] = mockQuery.mock.calls[0]; + expect(sql).toContain('ON CONFLICT (id) DO UPDATE'); + }); + }); + + describe('reject', () => { + it('persists rejection payload onto data JSONB', async () => { + const rejection = { + reason: 'fails policy', + timestamp: new Date('2026-05-11T00:00:00Z'), + reviewer: { username: 'r', reviewerEmail: 'r@example.com' }, + }; + + // First call: locked read of the row inside the transaction. + // Second call: the audit upsert on the same client. + mockQuery + .mockResolvedValueOnce({ + rowCount: 1, + rows: [{ data: { id: 'p1', authorised: false, canceled: false, rejected: false } }], + }) + .mockResolvedValueOnce({ rowCount: 1, rows: [] }); + + const result = await reject('p1', rejection as never); + + expect(result).toEqual({ message: 'reject p1' }); + + // The read must take a row lock so concurrent decisions serialise. + expect(String(mockQuery.mock.calls[0][0])).toContain('FOR UPDATE'); + + // The upsert call serializes the action (with rejection assigned) into + // the final query parameter as JSON text. + const upsertParams = mockQuery.mock.calls[1][1] as unknown[]; + const dataJson = JSON.parse(upsertParams[9] as string); + expect(dataJson).toMatchObject({ + id: 'p1', + rejected: true, + authorised: false, + canceled: false, + rejection: { reason: 'fails policy' }, + }); + }); + + it('throws if push is not found', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await expect(reject('missing', {} as never)).rejects.toThrow('push missing not found'); + }); + }); + + describe('authorise', () => { + it('marks the push authorised and clears canceled/rejected', async () => { + mockQuery + .mockResolvedValueOnce({ + rowCount: 1, + rows: [{ data: { id: 'p1', authorised: false, canceled: true, rejected: true } }], + }) + .mockResolvedValueOnce({ rowCount: 1, rows: [] }); + + const result = await authorise('p1', { token: 't' } as never); + + expect(result).toEqual({ message: 'authorised p1' }); + const upsertParams = mockQuery.mock.calls[1][1] as unknown[]; + const dataJson = JSON.parse(upsertParams[9] as string); + expect(dataJson).toMatchObject({ + id: 'p1', + authorised: true, + canceled: false, + rejected: false, + }); + }); + + it('throws if push is not found', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await expect(authorise('missing')).rejects.toThrow('push missing not found'); + }); + }); + + describe('cancel', () => { + it('marks the push canceled and clears authorised/rejected', async () => { + mockQuery + .mockResolvedValueOnce({ + rowCount: 1, + rows: [{ data: { id: 'p1', authorised: true, canceled: false, rejected: false } }], + }) + .mockResolvedValueOnce({ rowCount: 1, rows: [] }); + + const result = await cancel('p1'); + + expect(result).toEqual({ message: 'canceled p1' }); + const upsertParams = mockQuery.mock.calls[1][1] as unknown[]; + const dataJson = JSON.parse(upsertParams[9] as string); + expect(dataJson).toMatchObject({ + id: 'p1', + canceled: true, + authorised: false, + rejected: false, + }); + }); + + it('throws if push is not found', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await expect(cancel('missing')).rejects.toThrow('push missing not found'); + }); + }); + + describe('deletePush', () => { + it('issues a DELETE by id', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await deletePush('p1'); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('DELETE FROM pushes WHERE id = $1'); + expect(params).toEqual(['p1']); + }); + }); + + describe('list projection', () => { + it('drops steps from list results but not from the detail view', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await getPushes({}); + await getPushesForUserProfile([], 'alice'); + await getPush('p1'); + + const [listSql] = mockQuery.mock.calls[0]; + const [profileSql] = mockQuery.mock.calls[1]; + const [detailSql] = mockQuery.mock.calls[2]; + expect(listSql).toContain("data - 'steps'"); + expect(profileSql).toContain("data - 'steps'"); + expect(detailSql).not.toContain("data - 'steps'"); + }); + }); + + describe('getPushesForUserProfile', () => { + it('matches the reviewer case-insensitively when there are no emails', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await getPushesForUserProfile([], 'Alice'); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain("data->'attestation'->'reviewer'->>'username'"); + expect(sql).toMatch(/ORDER BY timestamp DESC/); + expect(sql).not.toContain('userEmail'); + expect(params).toEqual(['Alice']); + }); + + it('matches either the author email variants or the reviewer', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await getPushesForUserProfile(['a@b.com', 'A@B.com'], 'alice'); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain("(data->>'userEmail') = ANY($2::text[])"); + expect(sql).toContain(' OR '); + expect(params).toEqual(['alice', ['a@b.com', 'A@B.com']]); + }); + + it('returns Action instances', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [{ data: { id: 'p1', url: 'https://github.com/a/b.git' } }], + }); + + const result = await getPushesForUserProfile([], 'alice'); + + expect(result).toHaveLength(1); + expect(result[0].id).toBe('p1'); + }); + }); + + describe('getRepoPushRollupsByCanonicalUrl', () => { + const row = (over: Record = {}) => ({ + url: 'https://github.com/finos/git-proxy.git', + error: false, + rejected: false, + canceled: false, + authorised: false, + blocked: true, + allow_push: false, + timestamp: 1000, + ...over, + }); + + it('only scans push rows', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + + await getRepoPushRollupsByCanonicalUrl(); + + const [sql] = mockQuery.mock.calls[0]; + expect(sql).toContain("WHERE type = 'push'"); + }); + + it('counts pushes per canonical url and tracks the latest timestamps', async () => { + mockQuery.mockResolvedValue({ + rowCount: 2, + rows: [row({ timestamp: 1000 }), row({ timestamp: 5000 })], + }); + + const { tabCounts, latestPushAtMs, latestPendingReviewAtMs } = + await getRepoPushRollupsByCanonicalUrl(); + + const [key] = [...tabCounts.keys()]; + expect(tabCounts.get(key)?.pending).toBe(2); + expect(latestPushAtMs.get(key)).toBe(5000); + expect(latestPendingReviewAtMs.get(key)).toBe(5000); + }); + + it('separates approved pushes from pending ones', async () => { + mockQuery.mockResolvedValue({ + rowCount: 2, + rows: [row(), row({ authorised: true, blocked: false, timestamp: 9000 })], + }); + + const { tabCounts, latestPendingReviewAtMs } = await getRepoPushRollupsByCanonicalUrl(); + const [key] = [...tabCounts.keys()]; + + expect(tabCounts.get(key)?.pending).toBe(1); + expect(tabCounts.get(key)?.approved).toBe(1); + // the approved push must not advance the pending-review timestamp + expect(latestPendingReviewAtMs.get(key)).toBe(1000); + }); + + it('skips rows with an unusable url', async () => { + mockQuery.mockResolvedValue({ rowCount: 2, rows: [row({ url: null }), row({ url: '' })] }); + + const { tabCounts } = await getRepoPushRollupsByCanonicalUrl(); + + expect(tabCounts.size).toBe(0); + }); + + it('parses BIGINT timestamps returned as strings', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [row({ timestamp: '4200' })] }); + + const { latestPushAtMs } = await getRepoPushRollupsByCanonicalUrl(); + const [key] = [...latestPushAtMs.keys()]; + + expect(latestPushAtMs.get(key)).toBe(4200); + }); + + it('ignores non-numeric timestamps', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [row({ timestamp: null })] }); + + const { tabCounts, latestPushAtMs } = await getRepoPushRollupsByCanonicalUrl(); + + expect(tabCounts.size).toBe(1); + expect(latestPushAtMs.size).toBe(0); + }); + }); +}); diff --git a/test/db/postgres/repo.integration.test.ts b/test/db/postgres/repo.integration.test.ts new file mode 100644 index 000000000..dc5425f5d --- /dev/null +++ b/test/db/postgres/repo.integration.test.ts @@ -0,0 +1,213 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect } from 'vitest'; +import { + createRepo, + getRepo, + getRepoByUrl, + getRepoById, + getRepos, + addUserCanPush, + addUserCanAuthorise, + removeUserCanPush, + removeUserCanAuthorise, + deleteRepo, +} from '../../../src/db/postgres/repo'; +import { query } from '../../../src/db/postgres/helper'; +import { Repo } from '../../../src/db/types'; + +const shouldRunPostgresTests = process.env.RUN_POSTGRES_TESTS === 'true'; + +const createTestRepo = (overrides: Partial = {}): Repo => { + const id = Date.now() + Math.floor(Math.random() * 10_000); + return new Repo( + overrides.project ?? 'test-project', + overrides.name ?? `repo-${id}`, + overrides.url ?? `https://github.com/test-project/repo-${id}.git`, + overrides.users ?? { canPush: [], canAuthorise: [] }, + ); +}; + +describe.runIf(shouldRunPostgresTests)('PostgreSQL Repo Integration Tests', () => { + describe('createRepo', () => { + it('persists the row and stamps a generated _id', async () => { + const repo = createTestRepo({ name: 'create-test', url: 'https://example.com/x.git' }); + const created = await createRepo(repo); + + expect(created._id).toBeDefined(); + expect(created._id).toMatch(/^[0-9a-f-]{36}$/i); + + const fromDb = await getRepoByUrl('https://example.com/x.git'); + expect(fromDb?.name).toBe('create-test'); + }); + }); + + describe('getRepo / getRepoByUrl / getRepoById', () => { + it('finds by name (lower-cased lookup)', async () => { + await createRepo(createTestRepo({ name: 'findme', url: 'https://example.com/findme.git' })); + const found = await getRepo('FINDME'); + expect(found?.name).toBe('findme'); + }); + + it('finds by url exactly', async () => { + const url = 'https://example.com/url-test.git'; + await createRepo(createTestRepo({ name: 'url-test', url })); + const found = await getRepoByUrl(url); + expect(found?.url).toBe(url); + }); + + it('finds by _id', async () => { + const created = await createRepo( + createTestRepo({ name: 'id-test', url: 'https://example.com/id-test.git' }), + ); + const fromDb = await getRepoById(created._id as string); + expect(fromDb?.url).toBe('https://example.com/id-test.git'); + }); + + it('returns null when nothing matches', async () => { + expect(await getRepo('does-not-exist')).toBeNull(); + expect(await getRepoByUrl('https://nope.example/x.git')).toBeNull(); + }); + }); + + describe('getRepos', () => { + it('returns the seeded repos', async () => { + await createRepo(createTestRepo({ name: 'list-1', url: 'https://example.com/l1.git' })); + await createRepo(createTestRepo({ name: 'list-2', url: 'https://example.com/l2.git' })); + + const all = await getRepos(); + const names = all.map((r) => r.name); + expect(names).toEqual(expect.arrayContaining(['list-1', 'list-2'])); + }); + }); + + describe('permission membership', () => { + it('starts with empty arrays', async () => { + const created = await createRepo( + createTestRepo({ name: 'perm-start', url: 'https://example.com/ps.git' }), + ); + const fromDb = await getRepoById(created._id as string); + expect(fromDb?.users.canPush).toEqual([]); + expect(fromDb?.users.canAuthorise).toEqual([]); + }); + + it('adds a user without duplication', async () => { + const created = await createRepo( + createTestRepo({ name: 'perm-add', url: 'https://example.com/pa.git' }), + ); + const id = created._id as string; + + await addUserCanPush(id, 'Alice'); + await addUserCanPush(id, 'alice'); // duplicate (after lower-casing) + + const fromDb = await getRepoById(id); + expect(fromDb?.users.canPush).toEqual(['alice']); + }); + + it('removes the last user, leaving an empty array (NOT null)', async () => { + const created = await createRepo( + createTestRepo({ name: 'perm-remove', url: 'https://example.com/pr.git' }), + ); + const id = created._id as string; + + await addUserCanPush(id, 'bob'); + await removeUserCanPush(id, 'bob'); + + const fromDb = await getRepoById(id); + // Same behavior as Mongo and NeDB + expect(fromDb?.users.canPush).toEqual([]); + expect(fromDb?.users.canPush).not.toBeNull(); + }); + + it('applies the same invariant to canAuthorise', async () => { + const created = await createRepo( + createTestRepo({ name: 'auth-remove', url: 'https://example.com/ar.git' }), + ); + const id = created._id as string; + + await addUserCanAuthorise(id, 'reviewer'); + await removeUserCanAuthorise(id, 'reviewer'); + + const fromDb = await getRepoById(id); + expect(fromDb?.users.canAuthorise).toEqual([]); + expect(fromDb?.users.canAuthorise).not.toBeNull(); + }); + + it('keeps other users intact when removing one', async () => { + const created = await createRepo( + createTestRepo({ name: 'multi-perm', url: 'https://example.com/mp.git' }), + ); + const id = created._id as string; + + await addUserCanPush(id, 'alice'); + await addUserCanPush(id, 'bob'); + await removeUserCanPush(id, 'alice'); + + const fromDb = await getRepoById(id); + expect(fromDb?.users.canPush).toEqual(['bob']); + }); + }); + + describe('deleteRepo', () => { + it('deletes by _id', async () => { + const created = await createRepo( + createTestRepo({ name: 'del', url: 'https://example.com/del.git' }), + ); + await deleteRepo(created._id as string); + expect(await getRepoById(created._id as string)).toBeNull(); + }); + }); + + describe('repo_users normalization (issue #1559)', () => { + it('stores permissions as rows in repo_users, with no JSONB column on repos', async () => { + const created = await createRepo( + createTestRepo({ name: 'norm', url: 'https://example.com/norm.git' }), + ); + const id = created._id as string; + await addUserCanPush(id, 'alice'); + await addUserCanAuthorise(id, 'reviewer'); + + const rows = await query<{ username: string; role: string }>( + `SELECT username, role FROM repo_users WHERE repo_id = $1 ORDER BY role, username`, + [id], + ); + expect(rows.rows).toEqual([ + { username: 'reviewer', role: 'canAuthorise' }, + { username: 'alice', role: 'canPush' }, + ]); + + // The legacy JSONB column was dropped by migration 4 (drop_repos_users_jsonb). + const cols = await query<{ column_name: string }>( + `SELECT column_name FROM information_schema.columns WHERE table_name = 'repos'`, + ); + expect(cols.rows.map((r) => r.column_name)).not.toContain('users'); + }); + + it('cascades repo_users rows when the repo is deleted', async () => { + const created = await createRepo( + createTestRepo({ name: 'cascade', url: 'https://example.com/cascade.git' }), + ); + const id = created._id as string; + await addUserCanPush(id, 'alice'); + + await deleteRepo(id); + + const rows = await query(`SELECT 1 FROM repo_users WHERE repo_id = $1`, [id]); + expect(rows.rowCount).toBe(0); + }); + }); +}); diff --git a/test/db/postgres/repo.test.ts b/test/db/postgres/repo.test.ts new file mode 100644 index 000000000..bd5caf3e5 --- /dev/null +++ b/test/db/postgres/repo.test.ts @@ -0,0 +1,404 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +const mockQuery = vi.fn(); + +vi.mock('../../../src/db/postgres/helper', () => ({ + query: mockQuery, + // Runs the callback with a client whose query records into the same mock, + // so tests assert the statement sequence; transactional semantics themselves + // are covered by the withTransaction tests in helper.test.ts. + withTransaction: (fn: (client: { query: typeof mockQuery }) => Promise) => + fn({ query: mockQuery }), +})); + +describe('PostgreSQL - Repo', async () => { + const { + getRepos, + getRepo, + getRepoById, + getRepoByUrl, + updateRepo, + createRepo, + addUserCanPush, + addUserCanAuthorise, + removeUserCanPush, + removeUserCanAuthorise, + deleteRepo, + } = await import('../../../src/db/postgres/repo'); + + beforeEach(() => { + vi.clearAllMocks(); + }); + + describe('getRepos', () => { + it('builds WHERE clauses and maps the join-aggregated rows', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [ + { + _id: 'r1', + project: 'finos', + name: 'git-proxy', + url: 'https://example.com/finos/git-proxy', + can_push: ['bob'], + can_authorise: [], + }, + ], + }); + + const repos = await getRepos({ + name: 'Git-Proxy', + project: 'finos', + url: 'https://example.com/finos/git-proxy', + }); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('LEFT JOIN repo_users'); + expect(sql).toContain('WHERE'); + expect(sql).toContain('r.name = $1'); + expect(sql).toContain('r.project = $2'); + expect(sql).toContain('r.url = $3'); + expect(params).toEqual(['git-proxy', 'finos', 'https://example.com/finos/git-proxy']); + expect(repos[0].users.canPush).toEqual(['bob']); + }); + + it('adds no filter clause but still groups when no query is supplied', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await getRepos(); + const [sql, params] = mockQuery.mock.calls[0]; + expect(params).toEqual([]); + expect(sql).not.toContain('r.name ='); + expect(sql).not.toContain('r.url ='); + expect(sql).toContain('GROUP BY'); + }); + }); + + describe('getRepoByUrl', () => { + it('returns null when no row matches', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + expect(await getRepoByUrl('https://missing')).toBeNull(); + }); + + it('maps the aggregated row when found', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [ + { + _id: 'r1', + project: 'p', + name: 'n', + url: 'https://example.com/p/n', + can_push: [], + can_authorise: ['amy'], + }, + ], + }); + const repo = await getRepoByUrl('https://example.com/p/n'); + expect(repo?.users.canAuthorise).toEqual(['amy']); + }); + }); + + describe('read normalization', () => { + it('returns empty arrays when the aggregated columns are null', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [ + { + _id: 'r-1', + project: 'p', + name: 'n', + url: 'https://example.com/p/n', + can_push: null, + can_authorise: null, + }, + ], + }); + + const repo = await getRepoById('r-1'); + expect(repo?.users.canPush).toEqual([]); + expect(repo?.users.canAuthorise).toEqual([]); + }); + + it('lower-cases the name on getRepo', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await getRepo('MixedCase'); + expect(mockQuery.mock.calls[0][1]).toEqual(['mixedcase']); + }); + }); + + describe('createRepo', () => { + it('inserts the repo row and stamps _id from RETURNING', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [{ _id: 'generated-uuid' }] }); + + const created = await createRepo({ + project: 'finos', + name: 'git-proxy', + url: 'https://github.com/finos/git-proxy.git', + users: { canPush: [], canAuthorise: [] }, + } as never); + + expect(created._id).toBe('generated-uuid'); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('INSERT INTO repos'); + expect(params.slice(0, 3)).toEqual([ + 'finos', + 'git-proxy', + 'https://github.com/finos/git-proxy.git', + ]); + // date_created and last_modified are stamped on create + expect(typeof params[3]).toBe('string'); + expect(params[4]).toBe(params[3]); + // No second call: empty permissions mean no repo_users inserts. + expect(mockQuery).toHaveBeenCalledTimes(1); + }); + + it('persists supplied permissions into repo_users', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [{ _id: 'r9' }] }); + + await createRepo({ + project: 'p', + name: 'n', + url: 'https://x/n.git', + users: { canPush: ['bob'], canAuthorise: ['amy'] }, + } as never); + + // one set-based insert per role, lowercasing in SQL via unnest + const inserts = mockQuery.mock.calls.filter(([sql]) => + /INSERT INTO repo_users/.test(String(sql)), + ); + expect(inserts).toHaveLength(2); + expect(String(inserts[0][0])).toContain('unnest'); + expect(String(inserts[0][0])).toContain('lower(u.username)'); + expect(inserts[0][1]).toEqual(['r9', 'canPush', ['bob']]); + expect(inserts[1][1]).toEqual(['r9', 'canAuthorise', ['amy']]); + }); + }); + + describe('add / remove user', () => { + it('addUserCanPush inserts a lower-cased canPush row, ignoring duplicates', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await addUserCanPush('r-1', 'Bob'); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('INSERT INTO repo_users'); + expect(sql).toContain('ON CONFLICT DO NOTHING'); + expect(params).toEqual(['r-1', 'bob', 'canPush']); + }); + + it('addUserCanAuthorise targets the canAuthorise role', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await addUserCanAuthorise('r-1', 'Amy'); + expect(mockQuery.mock.calls[0][1]).toEqual(['r-1', 'amy', 'canAuthorise']); + }); + + it('removeUserCanPush deletes the lower-cased canPush row', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await removeUserCanPush('r-1', 'Bob'); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('DELETE FROM repo_users'); + expect(params).toEqual(['r-1', 'bob', 'canPush']); + }); + + it('removeUserCanAuthorise deletes the canAuthorise row', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await removeUserCanAuthorise('r-1', 'Amy'); + expect(mockQuery.mock.calls[0][1]).toEqual(['r-1', 'amy', 'canAuthorise']); + }); + }); + + describe('deleteRepo', () => { + it('issues a DELETE by _id (repo_users cascades)', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await deleteRepo('r1'); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('DELETE FROM repos WHERE _id = $1'); + expect(params).toEqual(['r1']); + }); + }); + + describe('updateRepo', () => { + it('writes only the supplied fields', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateRepo({ _id: 'r1', name: 'renamed' }); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('UPDATE repos SET name = $1'); + expect(sql).toContain('WHERE _id = $2'); + expect(params).toEqual(['renamed', 'r1']); + }); + + it('replaces permissions in the repo_users join table', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateRepo({ _id: 'r1', users: { canPush: ['alice'], canAuthorise: ['bob'] } }); + + const statements = mockQuery.mock.calls.map(([sql]) => sql); + // old rows are cleared first, then each role is re-inserted + expect(statements[0]).toContain('DELETE FROM repo_users WHERE repo_id = $1'); + const roles = mockQuery.mock.calls + .filter(([sql]) => /INSERT INTO repo_users/.test(String(sql))) + .map(([, params]) => params?.[1]); + expect(roles).toEqual(['canPush', 'canAuthorise']); + }); + + it('updates columns and permissions together', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateRepo({ _id: 'r1', name: 'renamed', users: { canPush: [], canAuthorise: [] } }); + + const statements = mockQuery.mock.calls.map(([sql]) => sql); + expect(statements[0]).toContain('UPDATE repos SET name = $1'); + expect(statements[1]).toContain('DELETE FROM repo_users'); + }); + + it('resets a field back to its column default when set to undefined', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateRepo({ _id: 'r1', project: undefined, name: 'keep' }); + + const [sql] = mockQuery.mock.calls[0]; + expect(sql).toContain('project = DEFAULT'); + expect(sql).toContain('name = $1'); + }); + + it('ignores unknown fields', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateRepo({ _id: 'r1', name: 'x', bogus: 'y' } as never); + + const [sql] = mockQuery.mock.calls[0]; + expect(sql).not.toContain('bogus'); + }); + + it('requires an _id', async () => { + await expect(updateRepo({ name: 'x' })).rejects.toThrow('updateRepo requires a repo _id'); + expect(mockQuery).not.toHaveBeenCalled(); + }); + + it('rejects an update with nothing to change', async () => { + await expect(updateRepo({ _id: 'r1' })).rejects.toThrow( + 'updateRepo requires at least one field to update', + ); + expect(mockQuery).not.toHaveBeenCalled(); + }); + }); + + describe('repo date fields', () => { + it('defaults dateCreated and lastModified on create', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [{ _id: 'r1' }] }); + + const repo = await createRepo({ + project: 'p', + name: 'n', + url: 'https://github.com/p/n.git', + } as never); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('date_created'); + expect(sql).toContain('last_modified'); + expect(repo.dateCreated).toBeTruthy(); + expect(repo.lastModified).toBe(repo.dateCreated); + expect(params[3]).toBe(repo.dateCreated); + }); + + it('keeps caller-supplied dates on create', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [{ _id: 'r1' }] }); + + const repo = await createRepo({ + project: 'p', + name: 'n', + url: 'https://github.com/p/n.git', + dateCreated: '2026-01-01T00:00:00.000Z', + lastModified: '2026-01-02T00:00:00.000Z', + } as never); + + expect(repo.dateCreated).toBe('2026-01-01T00:00:00.000Z'); + expect(repo.lastModified).toBe('2026-01-02T00:00:00.000Z'); + }); + + it('updateRepo writes dateCreated and lastModified columns', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateRepo({ + _id: 'r1', + dateCreated: '2026-01-01T00:00:00.000Z', + lastModified: '2026-01-01T00:00:00.000Z', + }); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('date_created = $1'); + expect(sql).toContain('last_modified = $2'); + expect(params).toEqual(['2026-01-01T00:00:00.000Z', '2026-01-01T00:00:00.000Z', 'r1']); + }); + + it('bumps last_modified when permissions change', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await addUserCanPush('r1', 'Alice'); + await removeUserCanPush('r1', 'Alice'); + + // each role change touches repo_users, then bumps last_modified on repos + const bumps = mockQuery.mock.calls.filter(([sql]) => sql.includes('SET last_modified = $2')); + expect(bumps).toHaveLength(2); + for (const [, params] of bumps) { + expect(params[0]).toBe('r1'); + expect(typeof params[1]).toBe('string'); + } + }); + + it('returns the date fields from reads', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [ + { + _id: 'r1', + project: 'p', + name: 'n', + url: 'u', + can_push: [], + can_authorise: [], + date_created: '2026-01-01T00:00:00.000Z', + last_modified: '2026-01-02T00:00:00.000Z', + }, + ], + }); + + const repos = await getRepos(); + + expect(repos[0].dateCreated).toBe('2026-01-01T00:00:00.000Z'); + expect(repos[0].lastModified).toBe('2026-01-02T00:00:00.000Z'); + }); + }); + + it('lowercases usernames in SQL when replacing permissions through updateRepo', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateRepo({ _id: 'r1', users: { canPush: ['Alice'], canAuthorise: ['BOB'] } }); + + // lowercasing happens in the statement itself (lower(u.username) over the + // unnested array), so the parameters carry the caller's original casing + const inserts = mockQuery.mock.calls.filter(([sql]) => + /INSERT INTO repo_users/.test(String(sql)), + ); + for (const [sql] of inserts) { + expect(String(sql)).toContain('lower(u.username)'); + } + expect(inserts.map(([, params]) => params?.[2])).toEqual([['Alice'], ['BOB']]); + }); +}); diff --git a/test/db/postgres/schemaMigrations.integration.test.ts b/test/db/postgres/schemaMigrations.integration.test.ts new file mode 100644 index 000000000..de9acfe1f --- /dev/null +++ b/test/db/postgres/schemaMigrations.integration.test.ts @@ -0,0 +1,221 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect } from 'vitest'; +import { Pool } from 'pg'; + +import { connect, query, resetConnection } from '../../../src/db/postgres/helper'; +import { + assertMigrationsCurrent, + MIGRATIONS, + runMigrations, +} from '../../../src/db/postgres/schemaMigrations'; + +const shouldRunPostgresTests = process.env.RUN_POSTGRES_TESTS === 'true'; + +// Mirrors the default in vitest.config.integration.postgres.ts. Used only by the +// backfill test below, which needs a raw pool to stage a pre-repo_users database. +const getConnectionString = () => + process.env.GIT_PROXY_POSTGRES_CONNECTION_STRING || + 'postgresql://postgres:postgres@localhost:5432/git_proxy_test'; + +const migration = (version: number) => { + const entry = MIGRATIONS.find((m) => m.version === version); + if (!entry) throw new Error(`migration ${version} not found`); + return entry; +}; + +// Every shipped version, in order — the expected schema_migrations content +// after a full run, without hardcoding the list length in each test. +const ALL_VERSIONS = MIGRATIONS.map((m) => m.version).sort((a, b) => a - b); + +// Drop everything so the next `connect()` exercises the migration runner from a +// genuinely empty database. The initial `query` self-bootstraps the schema; the +// DROP then clears it, and `resetConnection` releases the once-per-process latch +// so the following `connect()` re-runs migrations. +const resetToEmptyDatabase = async () => { + await query('DROP TABLE IF EXISTS schema_migrations, repo_users, pushes, repos, users CASCADE'); + await resetConnection(); +}; + +describe.runIf(shouldRunPostgresTests)('PostgreSQL Schema Migration Integration Tests', () => { + it('creates schema_migrations and the app tables and records version 1', async () => { + await resetToEmptyDatabase(); + + // First pool acquisition triggers the migration runner. + await connect(); + + const versions = await query<{ version: number }>( + 'SELECT version FROM schema_migrations ORDER BY version', + ); + expect(versions.rows.map((row) => row.version)).toEqual(ALL_VERSIONS); + + const tables = await query<{ tablename: string }>( + `SELECT tablename FROM pg_tables + WHERE schemaname = 'public' AND tablename IN ('users', 'repos', 'pushes', 'repo_users')`, + ); + expect(tables.rows.map((row) => row.tablename).sort()).toEqual([ + 'pushes', + 'repo_users', + 'repos', + 'users', + ]); + }); + + it('upgrades the users table to the version 2 shape on a fresh database', async () => { + await resetToEmptyDatabase(); + await connect(); + + const emailColumn = await query<{ is_nullable: string }>( + `SELECT is_nullable FROM information_schema.columns + WHERE table_name = 'users' AND column_name = 'email'`, + ); + expect(emailColumn.rows[0].is_nullable).toBe('YES'); + + const publicKeysColumn = await query<{ data_type: string }>( + `SELECT data_type FROM information_schema.columns + WHERE table_name = 'users' AND column_name = 'public_keys'`, + ); + expect(publicKeysColumn.rows[0].data_type).toBe('jsonb'); + }); + + it('creates the connect-pg-simple session table via the migration list', async () => { + await resetToEmptyDatabase(); + await connect(); + + const table = await query( + `SELECT 1 FROM pg_tables WHERE schemaname = 'public' AND tablename = 'session'`, + ); + expect(table.rowCount).toBe(1); + }); + + it('assertMigrationsCurrent fails on an empty database and passes after migrating', async () => { + const pool = new Pool({ connectionString: getConnectionString() }); + try { + await resetToEmptyDatabase(); + await expect(assertMigrationsCurrent(pool)).rejects.toThrow(/pending migrations/); + + // connect() runs every migration; the verification then finds nothing pending. + await connect(); + await expect(assertMigrationsCurrent(pool)).resolves.toBeUndefined(); + } finally { + await pool.end(); + } + }); + + it('is idempotent — re-running migrations does not duplicate the version row', async () => { + await connect(); + await resetConnection(); + await connect(); + + const versions = await query<{ version: number }>('SELECT version FROM schema_migrations'); + expect(versions.rows.map((row) => row.version)).toEqual(ALL_VERSIONS); + }); + + it('backfills existing JSONB repo permissions into repo_users (single, multi, same user in both roles)', async () => { + // A raw pool lets us stage the pre-repo_users state: apply v1+v2 by hand and + // record them as applied so `repos` still carries the legacy `users` JSONB + // column, then seed data, then let the runner apply only v3 (create + + // backfill) and v4 (drop column). Going through `connect()` instead would run + // every migration up front and drop the column before we could seed it. + const pool = new Pool({ connectionString: getConnectionString() }); + try { + await pool.query( + 'DROP TABLE IF EXISTS schema_migrations, repo_users, pushes, repos, users CASCADE', + ); + await pool.query(` + CREATE TABLE schema_migrations ( + version INTEGER PRIMARY KEY, + name TEXT NOT NULL, + applied_at TIMESTAMPTZ NOT NULL DEFAULT now() + )`); + for (const version of [1, 2]) { + const m = migration(version); + await pool.query(m.sql); + await pool.query('INSERT INTO schema_migrations (version, name) VALUES ($1, $2)', [ + m.version, + m.name, + ]); + } + + // Seed legacy repos carrying populated JSONB permissions. + const seed = async (name: string, users: { canPush: string[]; canAuthorise: string[] }) => + ( + await pool.query<{ _id: string }>( + 'INSERT INTO repos (name, url, users) VALUES ($1, $2, $3::jsonb) RETURNING _id', + [name, `https://example.com/${name}.git`, JSON.stringify(users)], + ) + ).rows[0]._id; + + const single = await seed('single', { canPush: ['alice'], canAuthorise: ['bob'] }); + const multi = await seed('multi', { canPush: ['amy', 'cara'], canAuthorise: ['dan'] }); + const both = await seed('both', { canPush: ['eve'], canAuthorise: ['eve'] }); + // Legacy entries can be mixed case; the runtime writers lowercase, so the + // backfill must too or these users become unretrievable. + const mixed = await seed('mixed', { canPush: ['Alice'], canAuthorise: ['ALICE', 'alice'] }); + const empty = await seed('empty', { canPush: [], canAuthorise: [] }); + + // Apply the remaining migrations: v3 creates repo_users + backfills, v4 + // drops the legacy column. The runner skips the already-recorded v1/v2. + await runMigrations(pool); + + const versions = await pool.query<{ version: number }>( + 'SELECT version FROM schema_migrations ORDER BY version', + ); + expect(versions.rows.map((r) => r.version)).toEqual(ALL_VERSIONS); + + // The legacy JSONB column is gone, dropped by v4. + const usersCol = await pool.query( + `SELECT 1 FROM information_schema.columns WHERE table_name = 'repos' AND column_name = 'users'`, + ); + expect(usersCol.rowCount).toBe(0); + + // Every JSONB entry was backfilled, one row per (repo, user, role). + const permsOf = async (id: string) => + ( + await pool.query<{ username: string; role: string }>( + 'SELECT username, role FROM repo_users WHERE repo_id = $1 ORDER BY role, username', + [id], + ) + ).rows; + + expect(await permsOf(single)).toEqual([ + { username: 'bob', role: 'canAuthorise' }, + { username: 'alice', role: 'canPush' }, + ]); + expect(await permsOf(multi)).toEqual([ + { username: 'dan', role: 'canAuthorise' }, + { username: 'amy', role: 'canPush' }, + { username: 'cara', role: 'canPush' }, + ]); + // A user listed in both roles becomes two PK-distinct rows. + expect(await permsOf(both)).toEqual([ + { username: 'eve', role: 'canAuthorise' }, + { username: 'eve', role: 'canPush' }, + ]); + + // Mixed-case entries are lowercased and case-only duplicates collapse. + expect(await permsOf(mixed)).toEqual([ + { username: 'alice', role: 'canAuthorise' }, + { username: 'alice', role: 'canPush' }, + ]); + // A repo with no permissions backfills nothing. + expect(await permsOf(empty)).toEqual([]); + } finally { + await pool.end(); + } + }); +}); diff --git a/test/db/postgres/schemaMigrations.test.ts b/test/db/postgres/schemaMigrations.test.ts new file mode 100644 index 000000000..f3496b0ef --- /dev/null +++ b/test/db/postgres/schemaMigrations.test.ts @@ -0,0 +1,194 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, vi } from 'vitest'; + +import { + assertMigrationsCurrent, + runMigrations, + MIGRATIONS, +} from '../../../src/db/postgres/schemaMigrations'; + +const SELECT_VERSIONS = /SELECT version FROM schema_migrations/; + +// Build a fake pg Pool whose single client records every query. `appliedRows` +// is what the `SELECT version FROM schema_migrations` lookup returns. +const makePool = (appliedRows: { version: number }[] = []) => { + const query = vi + .fn() + .mockImplementation((sql: string) => + SELECT_VERSIONS.test(sql) + ? Promise.resolve({ rows: appliedRows, rowCount: appliedRows.length }) + : Promise.resolve({ rows: [], rowCount: 0 }), + ); + const release = vi.fn(); + const pool = { connect: vi.fn().mockResolvedValue({ query, release }) }; + return { pool, query, release }; +}; + +const sqlsOf = (query: ReturnType) => query.mock.calls.map((call) => String(call[0])); + +describe('PostgreSQL - migrations', () => { + it('defines the pushes hot-path indexes as version 6', () => { + const v6 = MIGRATIONS.find((m) => m.version === 6); + expect(v6?.name).toBe('pushes_hot_path_indexes'); + expect(v6?.sql).toContain('pushes_rollup_idx'); + expect(v6?.sql).toContain( + 'INCLUDE (error, rejected, canceled, authorised, blocked, allow_push)', + ); + expect(v6?.sql).toContain('pushes_pending_idx'); + expect(v6?.sql).toContain('pushes_user_email_idx'); + expect(v6?.sql).toContain('pushes_reviewer_idx'); + }); + + it('exposes an ordered, append-only migration list starting at version 1', () => { + expect(MIGRATIONS[0].version).toBe(1); + + const versions = MIGRATIONS.map((m) => m.version); + expect(versions).toEqual([...versions].sort((a, b) => a - b)); + expect(new Set(versions).size).toBe(versions.length); + }); + + it('locks, then creates schema_migrations, then commits — in that order', async () => { + const { pool, query, release } = makePool([]); + + await runMigrations(pool as never); + + const sqls = sqlsOf(query); + expect(sqls[0]).toBe('BEGIN'); + expect(sqls[1]).toMatch(/pg_advisory_xact_lock/); + expect(sqls[2]).toMatch(/CREATE TABLE IF NOT EXISTS schema_migrations/); + expect(sqls[sqls.length - 1]).toBe('COMMIT'); + expect(release).toHaveBeenCalledTimes(1); + }); + + it('applies every pending migration and records its version', async () => { + const { pool, query } = makePool([]); + + await runMigrations(pool as never); + + const inserts = query.mock.calls.filter((call) => + /INSERT INTO schema_migrations/.test(String(call[0])), + ); + expect(inserts).toHaveLength(MIGRATIONS.length); + expect(inserts[0][1]).toEqual([MIGRATIONS[0].version, MIGRATIONS[0].name]); + + // The migration body runs before its bookkeeping insert. + const sqls = sqlsOf(query); + expect(sqls).toContain(MIGRATIONS[0].sql); + }); + + it('skips migrations already recorded as applied', async () => { + const allApplied = MIGRATIONS.map((m) => ({ version: m.version })); + const { pool, query } = makePool(allApplied); + + await runMigrations(pool as never); + + const inserts = query.mock.calls.filter((call) => + /INSERT INTO schema_migrations/.test(String(call[0])), + ); + expect(inserts).toHaveLength(0); + expect(sqlsOf(query)).toContain('COMMIT'); + }); + + it('rolls back and releases the client when a migration fails', async () => { + const query = vi.fn().mockImplementation((sql: string) => { + if (SELECT_VERSIONS.test(sql)) return Promise.resolve({ rows: [], rowCount: 0 }); + if (sql === MIGRATIONS[0].sql) return Promise.reject(new Error('migration boom')); + return Promise.resolve({ rows: [], rowCount: 0 }); + }); + const release = vi.fn(); + const pool = { connect: vi.fn().mockResolvedValue({ query, release }) }; + + await expect(runMigrations(pool as never)).rejects.toThrow('migration boom'); + + expect(sqlsOf(query)).toContain('ROLLBACK'); + expect(release).toHaveBeenCalledTimes(1); + }); + + it('rethrows the original error even when ROLLBACK also fails', async () => { + const query = vi.fn().mockImplementation((sql: string) => { + if (SELECT_VERSIONS.test(sql)) return Promise.resolve({ rows: [], rowCount: 0 }); + if (sql === MIGRATIONS[0].sql) return Promise.reject(new Error('migration boom')); + if (sql === 'ROLLBACK') return Promise.reject(new Error('rollback boom')); + return Promise.resolve({ rows: [], rowCount: 0 }); + }); + const release = vi.fn(); + const pool = { connect: vi.fn().mockResolvedValue({ query, release }) }; + + // The migration failure must surface, not the secondary rollback failure. + await expect(runMigrations(pool as never)).rejects.toThrow('migration boom'); + expect(release).toHaveBeenCalledTimes(1); + }); + + it('lowercases usernames in the repo_users backfill to match the runtime writers', () => { + const v4 = MIGRATIONS.find((m) => m.version === 4); + expect(v4?.sql).toContain('lower(elem.username)'); + }); + + it('owns the connect-pg-simple session table as version 7', () => { + const v7 = MIGRATIONS.find((m) => m.version === 7); + expect(v7?.name).toBe('session_table'); + // IF NOT EXISTS adopts databases where the store already created the table. + expect(v7?.sql).toContain('CREATE TABLE IF NOT EXISTS "session"'); + expect(v7?.sql).toContain('IDX_session_expire'); + }); + + describe('assertMigrationsCurrent', () => { + const makeCheckPool = (tableOid: string | null, appliedRows: { version: number }[]) => { + const query = vi.fn().mockImplementation((sql: string) => { + if (/to_regclass/.test(sql)) { + return Promise.resolve({ rows: [{ table_oid: tableOid }], rowCount: 1 }); + } + if (SELECT_VERSIONS.test(sql)) { + return Promise.resolve({ rows: appliedRows, rowCount: appliedRows.length }); + } + return Promise.reject(new Error(`unexpected statement: ${sql}`)); + }); + return { pool: { query }, query }; + }; + + it('passes silently when every migration is recorded', async () => { + const { pool } = makeCheckPool( + 'schema_migrations', + MIGRATIONS.map((m) => ({ version: m.version })), + ); + + await expect(assertMigrationsCurrent(pool as never)).resolves.toBeUndefined(); + }); + + it('names the pending versions when the schema is behind', async () => { + const allButLast = MIGRATIONS.slice(0, -1).map((m) => ({ version: m.version })); + const last = MIGRATIONS[MIGRATIONS.length - 1]; + const { pool } = makeCheckPool('schema_migrations', allButLast); + + await expect(assertMigrationsCurrent(pool as never)).rejects.toThrow( + new RegExp(`pending migrations: ${last.version} \\(${last.name}\\)`), + ); + }); + + it('treats a database without the bookkeeping table as fully pending, without DDL', async () => { + const { pool, query } = makeCheckPool(null, []); + + await expect(assertMigrationsCurrent(pool as never)).rejects.toThrow(/autoMigrate/); + + // Only the existence probe ran — never a SELECT against the missing + // table, and no CREATE of any kind. + expect(query).toHaveBeenCalledTimes(1); + expect(String(query.mock.calls[0][0])).toContain('to_regclass'); + }); + }); +}); diff --git a/test/db/postgres/users.integration.test.ts b/test/db/postgres/users.integration.test.ts new file mode 100644 index 000000000..48257699e --- /dev/null +++ b/test/db/postgres/users.integration.test.ts @@ -0,0 +1,319 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect } from 'vitest'; +import { + createUser, + findUser, + findUserByEmail, + findUserByGitAccount, + findUserByOIDC, + findUserBySSHKey, + getUsers, + updateUser, + deleteUser, + addPublicKey, + removePublicKey, + getPublicKeys, +} from '../../../src/db/postgres/users'; +import { DuplicateSSHKeyError } from '../../../src/errors/DatabaseErrors'; +import { PublicKeyRecord, User } from '../../../src/db/types'; + +const shouldRunPostgresTests = process.env.RUN_POSTGRES_TESTS === 'true'; + +describe.runIf(shouldRunPostgresTests)('PostgreSQL Users Integration Tests', () => { + const createTestUser = (overrides: Partial = {}): User => { + const timestamp = Date.now(); + return new User( + overrides.username || `testuser-${timestamp}`, + overrides.password || 'hashedpassword123', + overrides.gitAccount || `git-${timestamp}`, + overrides.email || `test-${timestamp}@example.com`, + overrides.admin ?? false, + overrides.oidcId || null, + ); + }; + + describe('createUser', () => { + it('lowercases username and email on insert', async () => { + const user = createTestUser({ username: 'CreateUser', email: 'Create@Example.COM' }); + await createUser(user); + + const found = await findUser('createuser'); + expect(found?.username).toBe('createuser'); + expect(found?.email).toBe('create@example.com'); + }); + }); + + describe('findUser', () => { + it('finds a user by username (case-insensitive)', async () => { + await createUser(createTestUser({ username: 'findme' })); + const result = await findUser('FINDME'); + expect(result?.username).toBe('findme'); + }); + + it('returns null for a non-existent user', async () => { + expect(await findUser('non-existent-user')).toBeNull(); + }); + }); + + describe('findUserByEmail', () => { + it('finds a user by email (case-insensitive)', async () => { + await createUser(createTestUser({ email: 'findbyemail@test.com' })); + const result = await findUserByEmail('FindByEmail@TEST.com'); + expect(result?.email).toBe('findbyemail@test.com'); + }); + + it('returns null for a non-existent email', async () => { + expect(await findUserByEmail('nonexistent@test.com')).toBeNull(); + }); + }); + + describe('findUserByGitAccount', () => { + it('finds a user by git account (case-insensitive), mirroring mongo', async () => { + await createUser(createTestUser({ username: 'gitacctuser', gitAccount: 'findbygit-acct' })); + const result = await findUserByGitAccount('FindByGit-Acct'); + expect(result?.gitAccount).toBe('findbygit-acct'); + }); + + it('returns null for a non-existent git account', async () => { + expect(await findUserByGitAccount('non-existent-git-account')).toBeNull(); + }); + }); + + describe('findUserByOIDC', () => { + it('finds a user by OIDC ID', async () => { + const oidcId = `oidc-${Date.now()}`; + await createUser(createTestUser({ oidcId })); + const result = await findUserByOIDC(oidcId); + expect(result?.oidcId).toBe(oidcId); + }); + + it('returns null for a non-existent OIDC ID', async () => { + expect(await findUserByOIDC('non-existent-oidc')).toBeNull(); + }); + }); + + describe('getUsers', () => { + it('retrieves users without their password', async () => { + await createUser(createTestUser({ username: 'getusers1' })); + await createUser(createTestUser({ username: 'getusers2' })); + + const result = await getUsers(); + + expect(result.length).toBeGreaterThanOrEqual(2); + result.forEach((user) => { + // Mirrors mongo's projection — passwords are null in list responses. + expect(user.password).toBeNull(); + }); + }); + + it('filters by username (lowercased)', async () => { + await createUser(createTestUser({ username: 'filteruser', email: 'filter@test.com' })); + await createUser(createTestUser({ username: 'otheruser', email: 'other@test.com' })); + + const result = await getUsers({ username: 'FilterUser' }); + + expect(result.length).toBe(1); + expect(result[0].username).toBe('filteruser'); + }); + + it('filters by email (lowercased)', async () => { + await createUser(createTestUser({ username: 'emailfilter', email: 'unique-email@test.com' })); + + const result = await getUsers({ email: 'Unique-Email@TEST.com' }); + + expect(result.length).toBe(1); + expect(result[0].email).toBe('unique-email@test.com'); + }); + }); + + describe('updateUser', () => { + it('updates by username and lowercases new fields', async () => { + await createUser(createTestUser({ username: 'updateme', admin: false })); + + await updateUser({ username: 'UpdateMe', admin: true }); + + const updated = await findUser('updateme'); + expect(updated?.admin).toBe(true); + }); + + it('updates by _id when provided', async () => { + await createUser(createTestUser({ username: 'updatebyid' })); + const created = await findUser('updatebyid'); + await updateUser({ _id: created?._id as string, gitAccount: 'new-git-account' }); + + const updated = await findUser('updatebyid'); + expect(updated?.gitAccount).toBe('new-git-account'); + }); + + it('lowercases email during update', async () => { + await createUser(createTestUser({ username: 'lowercaseupdate' })); + await updateUser({ username: 'LowerCaseUpdate', email: 'NEW@EMAIL.COM' }); + + const updated = await findUser('lowercaseupdate'); + expect(updated?.email).toBe('new@email.com'); + }); + + it('inserts when no row matches and only username is provided', async () => { + await updateUser({ + username: 'brand-new-user', + email: 'brand-new@example.com', + gitAccount: 'brand-new-git', + }); + + const inserted = await findUser('brand-new-user'); + expect(inserted?.email).toBe('brand-new@example.com'); + expect(inserted?.gitAccount).toBe('brand-new-git'); + }); + + it('allows multiple users without an email', async () => { + // e.g. users synced from AD, where the mail attribute is optional + await updateUser({ username: 'no-email-1', gitAccount: 'git-1' }); + await updateUser({ username: 'no-email-2', gitAccount: 'git-2' }); + + expect((await findUser('no-email-1'))?.username).toBe('no-email-1'); + expect((await findUser('no-email-2'))?.username).toBe('no-email-2'); + }); + + it('still rejects a duplicate non-empty email', async () => { + await createUser(createTestUser({ username: 'emailowner', email: 'taken@example.com' })); + + await expect( + createUser(createTestUser({ username: 'emailthief', email: 'taken@example.com' })), + ).rejects.toThrow(/duplicate key/); + }); + }); + + describe('deleteUser', () => { + it('deletes a user by username (case-insensitive)', async () => { + await createUser(createTestUser({ username: 'deleteme' })); + await deleteUser('DeleteMe'); + expect(await findUser('deleteme')).toBeNull(); + }); + }); + + describe('SSH public keys', () => { + const makeKey = (suffix: string): PublicKeyRecord => ({ + key: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5-${suffix}`, + name: `key-${suffix}`, + addedAt: new Date().toISOString(), + fingerprint: `SHA256:${suffix}`, + }); + + it('starts with an empty publicKeys array', async () => { + await createUser(createTestUser({ username: 'sshempty' })); + await expect(getPublicKeys('sshempty')).resolves.toEqual([]); + }); + + it('adds a key and finds the user by it', async () => { + const key = makeKey('add-and-find'); + await createUser(createTestUser({ username: 'sshadd' })); + await addPublicKey('sshadd', key); + + await expect(getPublicKeys('sshadd')).resolves.toEqual([key]); + const found = await findUserBySSHKey(key.key); + expect(found?.username).toBe('sshadd'); + }); + + it('rejects a key already registered to another user', async () => { + const key = makeKey('cross-user'); + await createUser(createTestUser({ username: 'sshowner' })); + await createUser(createTestUser({ username: 'sshthief' })); + await addPublicKey('sshowner', key); + + await expect(addPublicKey('sshthief', key)).rejects.toThrow(DuplicateSSHKeyError); + }); + + it('rejects a duplicate key for the same user', async () => { + const key = makeKey('same-user-dup'); + await createUser(createTestUser({ username: 'sshdup' })); + await addPublicKey('sshdup', key); + + await expect(addPublicKey('sshdup', key)).rejects.toThrow('SSH key already exists'); + }); + + it('rejects adding a key for a missing user', async () => { + await expect(addPublicKey('ssh-ghost', makeKey('ghost'))).rejects.toThrow('User not found'); + }); + + it('serialises concurrent adds of the same key: exactly one wins', async () => { + // Without the advisory lock inside addPublicKey, both calls can pass the + // duplicate check before either commits and the key ends up on two + // users. With it, the loser waits and then sees the winner's row. + const key = makeKey('race'); + await createUser(createTestUser({ username: 'sshracer1' })); + await createUser(createTestUser({ username: 'sshracer2' })); + + const results = await Promise.allSettled([ + addPublicKey('sshracer1', key), + addPublicKey('sshracer2', key), + ]); + + const fulfilled = results.filter((r) => r.status === 'fulfilled'); + const rejected = results.filter((r) => r.status === 'rejected'); + expect(fulfilled).toHaveLength(1); + expect(rejected).toHaveLength(1); + expect((rejected[0] as PromiseRejectedResult).reason).toBeInstanceOf(DuplicateSSHKeyError); + + // The key must belong to exactly one of the two users. + const keys1 = await getPublicKeys('sshracer1'); + const keys2 = await getPublicKeys('sshracer2'); + expect(keys1.length + keys2.length).toBe(1); + }); + + it('removes a key by fingerprint and leaves the rest', async () => { + const keep = makeKey('keep'); + const drop = makeKey('drop'); + await createUser(createTestUser({ username: 'sshremove' })); + await addPublicKey('sshremove', keep); + await addPublicKey('sshremove', drop); + + await removePublicKey('sshremove', drop.fingerprint); + + await expect(getPublicKeys('sshremove')).resolves.toEqual([keep]); + expect(await findUserBySSHKey(drop.key)).toBeNull(); + }); + + it('keeps an empty array (not null) after the last key is removed', async () => { + const key = makeKey('last-key'); + await createUser(createTestUser({ username: 'sshlast' })); + await addPublicKey('sshlast', key); + await removePublicKey('sshlast', key.fingerprint); + + await expect(getPublicKeys('sshlast')).resolves.toEqual([]); + }); + + it('is a no-op when removing an unknown fingerprint', async () => { + const key = makeKey('stable'); + await createUser(createTestUser({ username: 'sshnoop' })); + await addPublicKey('sshnoop', key); + + await removePublicKey('sshnoop', 'SHA256:does-not-exist'); + + await expect(getPublicKeys('sshnoop')).resolves.toEqual([key]); + }); + + it('round-trips publicKeys through createUser', async () => { + const key = makeKey('roundtrip'); + const user = createTestUser({ username: 'sshseeded' }); + user.publicKeys = [key]; + await createUser(user); + + await expect(getPublicKeys('sshseeded')).resolves.toEqual([key]); + }); + }); +}); diff --git a/test/db/postgres/users.test.ts b/test/db/postgres/users.test.ts new file mode 100644 index 000000000..ff9080b5c --- /dev/null +++ b/test/db/postgres/users.test.ts @@ -0,0 +1,360 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +const mockQuery = vi.fn(); + +vi.mock('../../../src/db/postgres/helper', () => ({ + query: mockQuery, + // Runs the callback with a client whose query records into the same mock, + // so tests assert the statement sequence; transactional semantics themselves + // are covered by the withTransaction tests in helper.test.ts. + withTransaction: (fn: (client: { query: typeof mockQuery }) => Promise) => + fn({ query: mockQuery }), +})); + +describe('PostgreSQL - Users', async () => { + const { + findUser, + findUserByEmail, + findUserByGitAccount, + findUserByOIDC, + findUserBySSHKey, + createUser, + deleteUser, + getUsers, + updateUser, + addPublicKey, + removePublicKey, + getPublicKeys, + } = await import('../../../src/db/postgres/users'); + + beforeEach(() => { + vi.clearAllMocks(); + }); + + describe('case insensitivity', () => { + it('lower-cases username on findUser', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await findUser('Mixed-Case'); + expect(mockQuery.mock.calls[0][1]).toEqual(['mixed-case']); + }); + + it('lower-cases email on findUserByEmail', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await findUserByEmail('USER@Example.COM'); + expect(mockQuery.mock.calls[0][1]).toEqual(['user@example.com']); + }); + + it('lower-cases gitAccount on findUserByGitAccount', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await findUserByGitAccount('Alice-Git'); + expect(mockQuery.mock.calls[0][1]).toEqual(['alice-git']); + }); + + it('lower-cases username/email on createUser', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await createUser({ + username: 'Alice', + password: 'pw', + gitAccount: 'alice-git', + email: 'Alice@Example.com', + admin: false, + } as never); + + const params = mockQuery.mock.calls[0][1] as unknown[]; + expect(params[0]).toBe('alice'); + expect(params[1]).toBe('alice@example.com'); + }); + + it('lower-cases username on deleteUser', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + await deleteUser('Alice'); + expect(mockQuery.mock.calls[0][1]).toEqual(['alice']); + }); + }); + + describe('row mapping', () => { + it('maps a DB row to a User on findUser', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [ + { + _id: 'u1', + username: 'alice', + email: 'alice@example.com', + password: 'hash', + git_account: 'alice-git', + admin: true, + oidc_id: null, + display_name: 'Alice A.', + title: 'Dev', + }, + ], + }); + + const user = await findUser('alice'); + + expect(user).toMatchObject({ + _id: 'u1', + username: 'alice', + email: 'alice@example.com', + password: 'hash', + gitAccount: 'alice-git', + admin: true, + displayName: 'Alice A.', + title: 'Dev', + }); + }); + }); + + describe('findUserByGitAccount', () => { + it('queries by git_account and returns null when absent', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + const user = await findUserByGitAccount('alice-git'); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('WHERE git_account = $1'); + expect(params).toEqual(['alice-git']); + expect(user).toBeNull(); + }); + }); + + describe('findUserByOIDC', () => { + it('queries by oidc_id and returns null when absent', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + const user = await findUserByOIDC('oidc-123'); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('WHERE oidc_id = $1'); + expect(params).toEqual(['oidc-123']); + expect(user).toBeNull(); + }); + }); + + describe('getUsers', () => { + it('omits password from the SELECT projection', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await getUsers({}); + const [sql] = mockQuery.mock.calls[0]; + expect(sql).toContain('NULL::text AS password'); + }); + + it('builds lower-cased username and email filters', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await getUsers({ username: 'Alice', email: 'Alice@Example.com' }); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('username = $1'); + expect(sql).toContain('email = $2'); + expect(params).toEqual(['alice', 'alice@example.com']); + }); + }); + + describe('updateUser', () => { + it('updates by _id when provided', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateUser({ _id: 'abc-123', displayName: 'Alice A.' } as never); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('UPDATE users SET'); + expect(sql).toContain('WHERE _id = $'); + expect(params).toEqual(['Alice A.', 'abc-123']); + }); + + it('upserts by username in a single atomic statement', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateUser({ username: 'new-user', email: 'new@example.com', admin: true } as never); + + expect(mockQuery).toHaveBeenCalledTimes(1); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('INSERT INTO users'); + expect(sql).toContain('ON CONFLICT (username) DO UPDATE SET'); + // Only the supplied fields are merged onto an existing row. + expect(sql).toContain( + 'username = EXCLUDED.username, email = EXCLUDED.email, admin = EXCLUDED.admin', + ); + expect(sql).not.toContain('password = EXCLUDED.password'); + // username is the first INSERT param. + expect((params as unknown[])[0]).toBe('new-user'); + }); + + it('lower-cases username and email in the upsert', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await updateUser({ username: 'ExistingUser', email: 'Updated@Example.com' } as never); + + const [, params] = mockQuery.mock.calls[0]; + expect((params as unknown[]).slice(0, 2)).toEqual(['existinguser', 'updated@example.com']); + }); + + it('throws if neither _id nor username is supplied', async () => { + await expect(updateUser({ admin: true } as never)).rejects.toThrow( + 'updateUser requires either _id or username', + ); + }); + + it('throws when no updatable field is supplied', async () => { + await expect(updateUser({ _id: 'abc-123' } as never)).rejects.toThrow( + 'updateUser requires at least one field to update', + ); + expect(mockQuery).not.toHaveBeenCalled(); + }); + }); + + describe('SSH public keys', () => { + const keyRecord = { + key: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA-test', + name: 'work laptop', + addedAt: '2026-01-01T00:00:00.000Z', + fingerprint: 'SHA256:abc123', + }; + + const userRow = (overrides: Record = {}) => ({ + _id: 'u1', + username: 'alice', + email: 'alice@example.com', + password: null, + git_account: 'alice-git', + admin: false, + oidc_id: null, + public_keys: [], + display_name: null, + title: null, + ...overrides, + }); + + describe('findUserBySSHKey', () => { + it('queries with JSONB containment on the key', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + const user = await findUserBySSHKey(keyRecord.key); + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain('public_keys @> $1::jsonb'); + expect(params).toEqual([JSON.stringify([{ key: keyRecord.key }])]); + expect(user).toBeNull(); + }); + + it('maps public_keys onto the returned User', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [userRow({ public_keys: [keyRecord] })], + }); + const user = await findUserBySSHKey(keyRecord.key); + expect(user?.publicKeys).toEqual([keyRecord]); + }); + }); + + describe('addPublicKey', () => { + // The first statement inside the transaction is the advisory lock that + // serialises concurrent adds of the same key. + const lockResult = { rowCount: 1, rows: [] }; + + it('appends the key to the user public_keys array', async () => { + mockQuery + .mockResolvedValueOnce(lockResult) // pg_advisory_xact_lock + .mockResolvedValueOnce({ rowCount: 0, rows: [] }) // duplicate-key check + .mockResolvedValueOnce({ rowCount: 1, rows: [userRow()] }) // locked user row + .mockResolvedValueOnce({ rowCount: 1, rows: [] }); // UPDATE + + await addPublicKey('Alice', keyRecord); + + expect(String(mockQuery.mock.calls[0][0])).toContain('pg_advisory_xact_lock'); + expect(mockQuery.mock.calls[0][1]).toEqual([keyRecord.key]); + expect(String(mockQuery.mock.calls[2][0])).toContain('FOR UPDATE'); + const [sql, params] = mockQuery.mock.calls[3]; + expect(sql).toContain('public_keys = public_keys || $2::jsonb'); + expect(params).toEqual(['alice', JSON.stringify([keyRecord])]); + }); + + it('throws DuplicateSSHKeyError when the key belongs to another user', async () => { + mockQuery.mockResolvedValueOnce(lockResult).mockResolvedValueOnce({ + rowCount: 1, + rows: [userRow({ username: 'bob', public_keys: [keyRecord] })], + }); + + await expect(addPublicKey('alice', keyRecord)).rejects.toThrow( + "SSH key already in use by user 'bob'", + ); + expect(mockQuery).toHaveBeenCalledTimes(2); + }); + + it('allows re-checking a key that already maps to the same user', async () => { + mockQuery.mockResolvedValueOnce(lockResult); + mockQuery.mockResolvedValueOnce({ + rowCount: 1, + rows: [userRow({ public_keys: [keyRecord] })], + }); + mockQuery.mockResolvedValueOnce({ + rowCount: 1, + rows: [userRow({ public_keys: [keyRecord] })], + }); + + await expect(addPublicKey('ALICE', keyRecord)).rejects.toThrow('SSH key already exists'); + }); + + it('throws when the user does not exist', async () => { + mockQuery + .mockResolvedValueOnce(lockResult) // pg_advisory_xact_lock + .mockResolvedValueOnce({ rowCount: 0, rows: [] }) // duplicate-key check + .mockResolvedValueOnce({ rowCount: 0, rows: [] }); // locked user row + + await expect(addPublicKey('ghost', keyRecord)).rejects.toThrow('User not found'); + }); + + it('throws when the fingerprint already exists for the user', async () => { + const existing = { ...keyRecord, key: 'ssh-ed25519 DIFFERENT-KEY' }; + mockQuery + .mockResolvedValueOnce(lockResult) + .mockResolvedValueOnce({ rowCount: 0, rows: [] }) + .mockResolvedValueOnce({ rowCount: 1, rows: [userRow({ public_keys: [existing] })] }); + + await expect(addPublicKey('alice', keyRecord)).rejects.toThrow('SSH key already exists'); + }); + }); + + describe('removePublicKey', () => { + it('filters the fingerprint out of public_keys and lower-cases username', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [] }); + + await removePublicKey('Alice', keyRecord.fingerprint); + + const [sql, params] = mockQuery.mock.calls[0]; + expect(sql).toContain(`(k->>'fingerprint') IS DISTINCT FROM $2`); + expect(params).toEqual(['alice', keyRecord.fingerprint]); + }); + }); + + describe('getPublicKeys', () => { + it('returns the user public keys', async () => { + mockQuery.mockResolvedValue({ + rowCount: 1, + rows: [userRow({ public_keys: [keyRecord] })], + }); + await expect(getPublicKeys('alice')).resolves.toEqual([keyRecord]); + }); + + it('returns [] when the column is null', async () => { + mockQuery.mockResolvedValue({ rowCount: 1, rows: [userRow({ public_keys: null })] }); + await expect(getPublicKeys('alice')).resolves.toEqual([]); + }); + + it('throws when the user does not exist', async () => { + mockQuery.mockResolvedValue({ rowCount: 0, rows: [] }); + await expect(getPublicKeys('ghost')).rejects.toThrow('User not found'); + }); + }); + }); +}); diff --git a/test/setup-integration-postgres.ts b/test/setup-integration-postgres.ts new file mode 100644 index 000000000..059b15737 --- /dev/null +++ b/test/setup-integration-postgres.ts @@ -0,0 +1,102 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { beforeAll, afterAll, afterEach } from 'vitest'; +import { Client } from 'pg'; + +import { resetConnection } from '../src/db/postgres/helper'; +import { invalidateCache } from '../src/config'; + +const DEFAULT_CONNECTION_STRING = 'postgresql://postgres:postgres@localhost:5432/git_proxy_test'; +// repo_users is listed before repos so the child table is cleaned first; the +// TRUNCATE/DROP CASCADE would handle the FK either way. +const APP_TABLES = ['repo_users', 'pushes', 'repos', 'users']; +const SESSION_TABLE = 'session'; +// Tracks applied schema versions. Persisted across tests (so the migration +// runner correctly skips already-applied versions) but dropped in afterAll so +// a re-run against the same database starts from a clean slate. +const MIGRATIONS_TABLE = 'schema_migrations'; + +let client: Client | null = null; + +const getConnectionString = () => + process.env.GIT_PROXY_POSTGRES_CONNECTION_STRING || DEFAULT_CONNECTION_STRING; + +const shouldRun = () => process.env.RUN_POSTGRES_TESTS === 'true'; + +beforeAll(async () => { + if (!shouldRun()) return; + + try { + client = new Client({ connectionString: getConnectionString() }); + await client.connect(); + console.log(`PostgreSQL connection established for integration tests`); + } catch (error) { + console.error('Failed to connect to PostgreSQL:', error); + throw error; + } +}); + +afterEach(async () => { + if (client) { + // Truncate app tables so each test starts from a known clean state. + // RESTART IDENTITY isn't needed (UUID PKs), but CASCADE keeps us future- + // proof in case a follow-up commit adds FK relationships. + try { + await client.query(`TRUNCATE TABLE ${APP_TABLES.join(', ')} CASCADE`); + } catch (error) { + console.warn('Failed to truncate app tables during integration test cleanup', error); + } + try { + // The session table is created lazily by connect-pg-simple; ignore the + // error if it does not yet exist. + await client.query(`TRUNCATE TABLE "${SESSION_TABLE}"`); + } catch { + // intentionally swallowed — table may not exist yet + } + } + + try { + await resetConnection(); + } catch (error) { + console.warn('Failed to reset Postgres pool during integration test cleanup', error); + } + invalidateCache(); +}); + +afterAll(async () => { + try { + await resetConnection(); + } catch (error) { + console.warn('Failed to reset Postgres pool during integration test cleanup', error); + } + + if (client) { + try { + for (const table of APP_TABLES) { + await client.query(`DROP TABLE IF EXISTS ${table} CASCADE`); + } + await client.query(`DROP TABLE IF EXISTS ${MIGRATIONS_TABLE} CASCADE`); + await client.query(`DROP TABLE IF EXISTS "${SESSION_TABLE}"`); + } catch (error) { + console.warn('Failed to drop Postgres test tables during cleanup', error); + } + await client.end(); + client = null; + } + + console.log('PostgreSQL integration test cleanup complete'); +}); diff --git a/vitest.config.integration.postgres.ts b/vitest.config.integration.postgres.ts new file mode 100644 index 000000000..773ca4848 --- /dev/null +++ b/vitest.config.integration.postgres.ts @@ -0,0 +1,43 @@ +/** + * Copyright 2026 GitProxy Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import path from 'path'; +import { defineConfig } from 'vitest/config'; + +export default defineConfig({ + test: { + include: ['test/db/postgres/**/*.integration.test.ts'], + testTimeout: 30000, + hookTimeout: 10000, + setupFiles: ['test/setup-integration-postgres.ts'], + pool: 'forks', + // The files share one database and some of them drop and recreate its + // tables, so they must not run concurrently. vitest 4 removed the old + // poolOptions.forks.singleFork switch (it silently no-ops), so file + // parallelism is disabled explicitly. + fileParallelism: false, + env: { + NODE_ENV: 'test', + RUN_POSTGRES_TESTS: 'true', + CONFIG_FILE: path.resolve(__dirname, 'test-integration.postgres.proxy.config.json'), + // Default for local runs; an exported GIT_PROXY_POSTGRES_CONNECTION_STRING + // (e.g. in CI or a non-default local setup) takes precedence. + GIT_PROXY_POSTGRES_CONNECTION_STRING: + process.env.GIT_PROXY_POSTGRES_CONNECTION_STRING || + 'postgresql://postgres:postgres@localhost:5432/git_proxy_test', + }, + }, +}); diff --git a/website/docs/architecture/architecture.md b/website/docs/architecture/architecture.md index 2efe2fb9c..5bdd004f4 100644 --- a/website/docs/architecture/architecture.md +++ b/website/docs/architecture/architecture.md @@ -484,10 +484,137 @@ Sample values: #### `sink` -List of database sources. The first source with `enabled` set to `true` will be used. Currently, MongoDB and filesystem databases ([NeDB](https://www.npmjs.com/package/@seald-io/nedb)) are supported. By default, the filesystem database is used. +List of database sources. The first source with `enabled` set to `true` will be used. GitProxy supports three sink backends: + +- **`fs`** — filesystem-backed [NeDB](https://www.npmjs.com/package/@seald-io/nedb). Default. Suitable for single-process deployments. +- **`mongo`** — MongoDB via `connect-mongo` for session storage. +- **`postgres`** — PostgreSQL via [`pg`](https://node-postgres.com/) + [`connect-pg-simple`](https://github.com/voxpelli/node-connect-pg-simple) for session storage. Each entry has its own unique configuration parameters. +##### PostgreSQL configuration + +The `postgres` backend stores `users`, `repos`, `pushes`, and the `connect-pg-simple` `session` table in a single PostgreSQL database. The required tables are created and kept up to date on startup by a built-in versioned migration runner (see [Schema migrations](#schema-migrations) below), so pointing the proxy at an empty database is enough to get running. + +```json +{ + "sink": [ + { + "type": "postgres", + "connectionString": "postgresql://user:pass@host:5432/gitproxy", + "enabled": true + } + ] +} +``` + +If `connectionString` is omitted on the config entry, GitProxy falls back to the `GIT_PROXY_POSTGRES_CONNECTION_STRING` environment variable. This mirrors the behaviour of the mongo backend's `GIT_PROXY_MONGO_CONNECTION_STRING`. + +##### Connection options + +Beyond `connectionString`, the `postgres` sink accepts discrete connection fields and tuning options: + +- `host`, `port`, `user`, `password`, `database` - used when `connectionString` is not set. +- `ssl` - `true` for TLS with default certificate verification, or an object of TLS options (`rejectUnauthorized`, `ca`, `cert`, `key`, ...). +- `pool` - pool tuning: `max`, `idleTimeoutMillis`, `connectionTimeoutMillis`. + +Connection precedence: `connectionString` (the config field, then `GIT_PROXY_POSTGRES_CONNECTION_STRING`) wins; otherwise the discrete fields are used; if neither is set, the standard `PGHOST` / `PGPORT` / `PGUSER` / `PGPASSWORD` / `PGDATABASE` environment variables are read by the client. `ssl` and `pool` are applied in all cases. If none of these resolve to a connection, GitProxy refuses to start rather than silently defaulting to `localhost`. + +```json +{ + "type": "postgres", + "host": "db.example.com", + "port": 5432, + "user": "gitproxy", + "password": "...", + "database": "gitproxy", + "ssl": { "rejectUnauthorized": true }, + "pool": { "max": 20, "idleTimeoutMillis": 30000 }, + "enabled": true +} +``` + +##### AWS RDS / Aurora IAM authentication + +For Amazon RDS or Aurora, GitProxy can authenticate with a short-lived IAM auth token instead of a static password. Enable `awsIamAuth` and supply the discrete `host` / `port` / `user` fields (a `connectionString` is not used in this mode): + +```json +{ + "type": "postgres", + "host": "mydb.abc123.eu-west-2.rds.amazonaws.com", + "port": 5432, + "user": "gitproxy_iam", + "database": "gitproxy", + "awsIamAuth": { "enabled": true, "region": "eu-west-2" }, + "enabled": true +} +``` + +- A fresh token is generated for every new pool connection from the AWS SDK default credential chain (via `@aws-sdk/rds-signer`), so no password is stored and token refresh is automatic. +- `region` falls back to the `AWS_REGION` / `AWS_DEFAULT_REGION` environment variables, then the SDK's default region resolution. +- TLS is required by RDS for IAM auth. Supply the RDS certificate authority bundle via `ssl` (for example `{ "rejectUnauthorized": true, "ca": "" }`, downloadable from https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem): RDS server certificates chain to Amazon's own root CA, which is not in Node's default trust store, so the `ssl: true` fallback (used when `ssl` is omitted) fails certificate verification against a real RDS endpoint and logs a startup warning saying so. Do not work around a verification failure with `rejectUnauthorized: false`; that discards the transport security IAM auth depends on. +- The database user must be granted the `rds_iam` role (`GRANT rds_iam TO gitproxy_iam;`). +- IAM auth needs the optional `@aws-sdk/rds-signer` dependency, which installs by default. On a slim install (`npm install --omit=optional`) add it explicitly with `npm install @aws-sdk/rds-signer`. + +##### Migrating data into PostgreSQL + +To copy existing `users`, `repos` and `pushes` from a `mongo` or `fs` (NeDB) backend into PostgreSQL, first switch the active sink to `postgres` (the destination), then run: + +```bash +# From MongoDB +npm run migrate:postgres -- --from mongo --mongoUrl "mongodb://user:pass@host:27017/git-proxy" + +# From the filesystem (NeDB) backend +npm run migrate:postgres -- --from fs --dataDir ./.data/db +``` + +The importer reads the source with its own driver while writing through the active (postgres) sink, so the two connections never clash. It is idempotent: users and repos that already exist (matched by username/email and URL) are skipped, and pushes are upserted by id, so it is safe to re-run. Record `_id`s are not carried over; PostgreSQL assigns fresh UUIDs (push ids, which are text, are preserved). + +##### Schema migrations + +Schema changes are applied by a small built-in migration runner (`src/db/postgres/schemaMigrations.ts`). On every startup it: + +- ensures a `schema_migrations` bookkeeping table exists, +- takes a transaction-scoped advisory lock so concurrently starting processes do not race, and +- applies any migrations whose version has not been recorded yet, in order, recording each as it goes. + +Migrations are an ordered, append-only list of SQL statements defined in code. Version 1 is the initial schema; because it uses `CREATE TABLE IF NOT EXISTS`, databases that were bootstrapped by earlier releases adopt the runner transparently (version 1 is simply recorded). To evolve the schema, append a new entry with the next version number; never edit or reorder migrations that have already shipped. + +Notes and current limitations: + +- All pending migrations run inside a single transaction, so a statement that cannot run transactionally (for example `CREATE INDEX CONCURRENTLY`) is not yet supported by the runner. +- Repo permissions (`canPush` / `canAuthorise`) are normalised into a `repo_users(repo_id, username, role)` join table (`ON DELETE CASCADE` from `repos`); the adapter reconstructs the permission arrays on read. +- The `connect-pg-simple` session table is created by the migration list too (rather than by the store's own `createTableIfMissing`), so every piece of DDL flows through the same versioned, locked runner. +- If `postgres` is selected as the active sink and no connection can be resolved, GitProxy refuses to start rather than silently falling back to an in-memory session store. + +###### Disabling automatic migrations (`autoMigrate: false`) + +Running migrations lazily at startup means the runtime database role permanently holds DDL rights. Where that is not acceptable — regulated deployments commonly separate DDL and DML credentials — set `"autoMigrate": false` on the postgres sink entry. Startup then only verifies that the schema is current, refusing to start (and naming the pending versions) when it is not, and migrations are applied out-of-band with DDL-capable credentials: + +```bash +npm run migrate:postgres:schema +``` + +The script connects using the configured sink (or the standard `PG*` / `GIT_PROXY_POSTGRES_CONNECTION_STRING` overrides, letting you substitute elevated credentials), applies any pending migrations under the same advisory lock as the startup path, and exits. + +###### Deploy ordering + +A migration can retire schema that an older, still-running GitProxy process depends on (migration 5, for instance, drops the legacy `repos.users` column that older processes read). When upgrading a multi-process deployment across such a migration, stop or fully drain the processes running the older version before the new version boots — or, with `autoMigrate` off, before running `migrate:postgres:schema`. A rolling deploy that applies migrations while old processes are still serving can break those processes mid-flight. + +##### PostgreSQL design decisions + +The adapter follows a few deliberate choices, made for parity with the existing backends rather than for idiomatic SQL: + +- **Pushes stay documents.** A push is an audit record: written once, updated through a handful of state flips, and read back whole. The `pushes` table therefore keeps the entire action as a JSONB `data` column, with typed columns (`timestamp`, the status booleans) only for the fields that queries filter and sort on. This mirrors how the mongo and NeDB backends treat pushes and keeps the row shape stable as the `Action` type evolves. +- **JSONB over full normalisation is deliberate.** Fully normalising an action would decompose a deeply nested document (steps, commit data, attestation) across many tables on every write and reassemble it with multi-way joins on every read, to serve relational queries the application never makes: pushes are fetched whole by id and listed by timestamp, and the few filtered fields are already real columns (with expression indexes covering the JSONB lookups the profile and activity pages make). Reads are single-row fetches either way, and a write is one upsert instead of a transactional multi-table write, so for this workload the document layout is at least as fast in both directions. It also keeps all three backends operating on the same document shapes, which is what makes feature parity across sinks tractable and lets `migrate:postgres` move data between backends without lossy transformation. Where the data _is_ queried relationally — repo permissions — the schema is normalised instead (`repo_users` above): JSONB is reserved for data that is genuinely a document. +- **Users and repos are typed rows with JSONB edges.** Fields that queries touch get real columns; genuinely document-shaped parts (a user's `publicKeys`) are JSONB, while repo permissions live in the normalised `repo_users` join table. +- **Identifiers are server-generated UUIDs** (`gen_random_uuid()`), the SQL analogue of mongo's ObjectIds. No compatibility between the backends' id formats is assumed anywhere in the app. +- **Timestamps the app treats as strings stay strings.** `dateCreated` and `lastModified` are ISO-8601 `TEXT` columns so values round-trip byte-for-byte identically to the mongo and NeDB backends, with no timezone conversion on the way through. +- **Same case rules as mongo**: usernames are lowercased on permission changes, and repo name lookups are lowercase. +- **Email uniqueness is best-effort**, enforced by a partial unique index: any number of users may have no email (the ActiveDirectory `mail` attribute is optional), while a real address can only be claimed once. This matches the permissive behaviour of the other backends. +- **Sessions use `connect-pg-simple`**, the postgres counterpart of the mongo backend's `connect-mongo` session store. +- **Failures are loud.** If `postgres` is the active sink and no connection can be resolved, GitProxy refuses to start rather than silently degrading to an in-memory session store. + Extending GitProxy to support other databases requires adding the relevant handlers and setup to the [`/src/db`](https://github.com/finos/git-proxy/blob/main/src/db/) directory. Feel free to [open an issue](https://github.com/finos/git-proxy/issues) requesting support for any specific databases - or [open a PR](https://github.com/finos/git-proxy/pulls) with the desired changes! #### `authentication`