feat: data migration from mongo/fs to the PostgreSQL sink - #1582
Conversation
✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## feat/postgres #1582 +/- ##
=================================================
+ Coverage 86.48% 86.64% +0.15%
=================================================
Files 108 111 +3
Lines 5986 6071 +85
Branches 1111 1124 +13
=================================================
+ Hits 5177 5260 +83
Misses 527 527
- Partials 282 284 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…gration Users synced from AD can lack both fields (the mail attribute is optional), and the postgres writers dereference them. Public keys now flow through unchanged: the sources read full documents and createUser persists user.publicKeys since the SSH support landed on the sink branch.
|
If you start working these PRs from FINOS repo branches (which you should be able to do as a member of the (trusted) contributors team) then you could target these PRs at the the other PR branch so we only see the diff for the additions... Once the the other PR merges and its branch is deleted github switches the target to main automatically. If you are using the DB adaptor to perform the migration (only took a brief look so far), can this not be generalised to handle migration to/from any adaptor? That would be preferable to a postgres as destination specific migration tool (I'm not convinced thats the right direction :-p ). |
|
Nice! |
# Conflicts: # website/docs/architecture/architecture.md
Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
jescalada
left a comment
jescalada
left a comment
There was a problem hiding this comment.
Thanks for this! Just a few comments on this one - and Fabio's comment to fix up 👍🏼
Review feedback on the data migration: - Pushes now stream through the migration in batches (default 500) instead of one toArray() call, so production tables with tens of thousands of diff-carrying documents no longer sit in memory whole. The mongo source iterates a cursor; the NeDB source chunks its in-memory records to the same contract. - The file source fails fast on a missing data directory or one that contains no fs sink datastores, and loads explicitly so a corrupt datastore surfaces as an error instead of reading as empty. - The CLI closes the destination pool after the summary, so the process exits instead of hanging on open handles. - Tests document that passwords are copied verbatim (hashing lives in the service wrapper, not the adapter) and that createRepo persists the permission map without separate role calls.
…-data-migration # Conflicts: # website/docs/architecture/architecture.md
jescalada
left a comment
jescalada
left a comment
There was a problem hiding this comment.
LGTM, just needs conficts fixed 🚀
…-data-migration # Conflicts: # website/docs/architecture/architecture.md
4 participants
Description
Adds a data-migration utility to copy
users,reposandpushesfrom amongoorfs(NeDB) backend into PostgreSQL.src/db/postgres/migrate.tsorchestrates the import. It is idempotent: users and repos that already exist (matched by username/email and URL) are skipped, and pushes are upserted by id, so the import is safe to re-run.migrateMongoSource.ts(MongoDB driver) andmigrateFileSource.ts(NeDB) read from the legacy backend with their own connection while the importer writes through the active PostgreSQL sink, so the two connections never clash._ids are not carried over; PostgreSQL assigns fresh UUIDs. Push ids (text) are preserved by the upsert.npm run migrate:postgres -- --from <mongo|fs> [--mongoUrl ... | --dataDir ...].Note: the integration test exercises the fs to postgres path only, because the PostgreSQL CI lane does not run a Mongo service. The mongo source is unit-tested with a mocked driver. Wiring a Mongo service into that lane for full mongo-to-postgres integration coverage is a reasonable follow-up.
Related Issue
Resolves #1557
Checklist
General
Documentation
Configuration
config.schema.json) was modified: (not modified in this PR)Tests
npm test)npm run lintandnpm run format:check)npm run check-types)