Delta Migration Feature#1044
Open
yashin4112 wants to merge 118 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Implements “delta migration” support by introducing an iteration counter, restart flow, and per-iteration mapping/UID persistence so subsequent runs can selectively recreate content types, deduplicate assets, and update existing entries.
Changes:
- Replaces upload-api runtime config import with a JSON-backed config file and adds a helper to refresh/update config at runtime.
- Adds “Restart Migration” UX + iteration tracking in UI (stepper/header/logs) and introduces an Entry Mapper table for iteration > 1.
- Adds API support for iteration-scoped LowDB files (content types, fields, entries, uid-mapper) plus delta helpers (asset dedup + entry update script + restart endpoint).
Reviewed changes
Copilot reviewed 50 out of 55 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| upload-api/src/services/fileProcessing.ts | Reads refreshed config during file validation/processing. |
| upload-api/src/services/aws/client.ts | Switches AWS config source to JSON config file. |
| upload-api/src/routes/index.ts | Refreshes config per request and returns sanitized config via /config. |
| upload-api/src/helper/index.ts | Adds updateConfigFile helper for JSON config read/write. |
| upload-api/src/controllers/sitecore/index.ts | Invokes Sitecore extractEntries during mapper creation. |
| upload-api/src/config/index.ts | Removes TS-based config module. |
| upload-api/src/config/index.json | Adds JSON-based config template. |
| upload-api/package.json | Formatting-only change. |
| upload-api/migration-wordpress/libs/extractEntries.ts | Adds WordPress entry extraction to build entryMapping. |
| upload-api/migration-sitecore/libs/extractEntries.js | Adds Sitecore entry extraction to build entryMapping. |
| upload-api/migration-sitecore/index.js | Exposes extractEntries from migration-sitecore package entrypoint. |
| upload-api/migration-drupal/libs/extractEntries.js | Adds Drupal DB entry extraction to build entryMapping. |
| upload-api/migration-drupal/libs/createInitialMapper.js | Attaches Drupal entryMapping into initial mapper output. |
| upload-api/migration-drupal/index.js | Exposes extractEntries from migration-drupal package entrypoint. |
| upload-api/migration-contentful/libs/extractEntries.js | Adds Contentful export entry extraction grouped by content type. |
| upload-api/migration-contentful/libs/createInitialMapper.js | Attaches Contentful entryMapping into initial mapper output. |
| upload-api/migration-contentful/index.js | Exposes extractEntries from migration-contentful package entrypoint. |
| upload-api/migration-aem/package-lock.json | Lockfile updates (types/undici-types). |
| ui/src/services/api/migration.service.ts | Adds restart + entry-mapping API calls for delta migration UI. |
| ui/src/pages/Migration/index.tsx | Adds “Restart Migration” execution path and iteration increment. |
| ui/src/context/app/app.interface.ts | Adds iteration and stepValue to migration state and defaults. |
| ui/src/components/Stepper/HorizontalStepper/HorizontalStepper.tsx | Resets stepper state on restart/iteration increment. |
| ui/src/components/MigrationFlowHeader/index.tsx | Derives CTA label (“Restart Migration”, etc.) into Redux state. |
| ui/src/components/LogScreen/MigrationLogViewer.tsx | Avoids duplicate completion notifications; sets “Restart Migration” CTA. |
| ui/src/components/LegacyCms/legacyCms.scss | Adds layout styles for editable local-path display. |
| ui/src/components/LegacyCms/Actions/LoadUploadFile.tsx | Adds inline editing UX for local path in restart iterations. |
| ui/src/components/ContentMapper/index.tsx | Switches to Entry Mapper view when iteration > 1; refetches on iteration change. |
| ui/src/components/ContentMapper/index.scss | Scopes table wrapper styles under .content-mapper-container. |
| ui/src/components/ContentMapper/entryMapper.tsx | New Entry Mapper table for selecting entries to update in delta runs. |
| ui/src/components/ContentMapper/contentMapper.interface.ts | Adds EntryMapperType interface for entry-mapping table rows. |
| ui/package.json | Formatting-only change. |
| package.json | Formatting-only change. |
| api/src/utils/package.json | Adds a package.json under src/utils (appears unrelated to runtime). |
| api/src/utils/field-attacher.utils.ts | Makes content type creation conditional based on iteration and prior existence. |
| api/src/utils/entry-update.utils.ts | Adds logic to remove/update entries and build update config for CLI script. |
| api/src/utils/entry-update-script.cjs | Adds CLI migration script to update existing entries (incl. asset resolution). |
| api/src/utils/entry-duplicate.utils.ts | Flags duplicate entries in entry mapper DB. |
| api/src/utils/content-type-checker.utils.ts | Adds helper to skip content type creation if created in prior iterations. |
| api/src/utils/asset-update.utils.ts | Adds asset dedup logic across iterations and ref replacement in entry JSON. |
| api/src/services/updateEntryCli.service.ts | Adds service to run CLI “update entries” script after delta import. |
| api/src/services/runCli.service.ts | Persists uid-mapping output into iteration-scoped LowDB (uid-mapper.json). |
| api/src/services/projects.service.ts | Stores iteration on project creation; uses iteration-scoped DBs on delete. |
| api/src/services/migration.service.ts | Adds restart endpoint + delta pre/post steps (asset dedup + entry updates). |
| api/src/services/contentMapper.service.ts | Adds iteration-scoped DB access + entry-mapping endpoints + uid enrichment. |
| api/src/routes/migration.routes.ts | Adds POST /restart/:orgId/:projectId route. |
| api/src/routes/contentMapper.routes.ts | Adds entry mapping GET + entry status update PUT routes. |
| api/src/models/uidMapper.ts | Adds iteration-scoped LowDB model for combined asset/entry uid mapping. |
| api/src/models/project-lowdb.ts | Adds iteration field to Project model. |
| api/src/models/contentTypesMapper-lowdb.ts | Converts content type mapper DB to iteration-scoped per-project storage. |
| api/src/models/FieldMapper.ts | Converts field mapper DB to iteration-scoped per-project storage. |
| api/src/models/EntryMapper.ts | Adds iteration-scoped LowDB model for entry mapper rows. |
| api/src/controllers/projects.contentMapper.controller.ts | Exposes entry mapping + update status endpoints from controller. |
| api/src/controllers/migration.controller.ts | Exposes restart migration endpoint from controller. |
| api/package.json | Bumps CLI dependencies for migration/update flows. |
Files not reviewed (1)
- upload-api/migration-aem/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… service and utils; unit test cases added
…t tests for content type creation logic
…sting data reference
…ping resolution in Contentful service
…esModal with async handling for step changes
…le package.json and package-lock.json files
…r better handling of missing and media blocks
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…iple package.json and package-lock.json files
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…iple package.json and package-lock.json files
…api and upload-api - Upgraded @contentstack/cli to version 1.61.1 in both api and upload-api. - Updated @wordpress/block-serialization-default-parser to version 5.46.0 in api. - Updated @wordpress/block-library to version 9.46.0 and @wordpress/blocks to version 15.19.0 in upload-api. - Bumped @babel/runtime to version 7.29.2 in upload-api. - Added new dependency @arraypress/waveform-player version 1.2.1 in upload-api.
…kage-lock.json - Added yaml package with version 2.4.2 to both package.json and package-lock.json. - Updated existing yaml dependency version from 1.10.3 to 2.9.0 in package-lock.json, including new metadata for funding and binary.
…iple package.json and package-lock.json files
…api and upload-api - Upgraded @contentstack/cli to version 1.61.1 in both api and upload-api. - Updated @wordpress/block-serialization-default-parser to version 5.46.0 in api. - Updated @wordpress/block-library to version 9.46.0 and @wordpress/blocks to version 15.19.0 in upload-api. - Bumped @babel/runtime to version 7.29.2 in upload-api. - Added new dependency @arraypress/waveform-player version 1.2.1 in upload-api.
…kage-lock.json - Added yaml package with version 2.4.2 to both package.json and package-lock.json. - Updated existing yaml dependency version from 1.10.3 to 2.9.0 in package-lock.json, including new metadata for funding and binary.
- Introduced a new script `hydrate-config.ts` to hydrate configuration from environment variables. - Updated `package.json` to include new npm scripts for hydration. - Refactored `index.ts` to call `hydrateConfig` during application startup. - Enhanced `index.ts` routes to support file uploads with Docker compatibility. - Implemented recursive directory copying in the upload route. - Added unit tests for new functionality in `hydrate-config` and routes.
…recursive copy safety
…vent directory traversal attacks
…vent directory traversal attacks
…iner route to prevent directory traversal vulnerabilities
…tion - Deleted the entire test suite for fetchMarketplaceInstallationsForStack in market-app.utils.test.ts. - Updated package-lock.json to mark several dependencies as dev dependencies and added new date-fns entries with funding information.
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.