Add runtime migration creation and application

[thromel]

Summary

Implements #37342: Allow creating and applying migrations at runtime without recompiling.

This adds support for creating and applying migrations at runtime using Roslyn compilation, enabling scenarios like .NET Aspire and containerized applications where recompilation isn't possible.

CLI Usage

# Standard update (existing behavior)
dotnet ef database update [migration]

# Create and apply a new migration in one step
dotnet ef database update MigrationName --add [--output-dir <DIR>] [--namespace <NS>] [--json]

The -o/--output-dir, -n/--namespace, and --json options require --add to be specified.

PowerShell Usage

# Standard update (existing behavior)
Update-Database [-Migration <migration>]

# Create and apply a new migration in one step
Update-Database -Migration MigrationName -Add [-OutputDir <DIR>] [-Namespace <NS>]

Architecture

Component	Purpose
IMigrationCompiler / CSharpMigrationCompiler	Internal: Roslyn-based compilation of scaffolded migrations
IMigrationsAssembly.AddMigrations(Assembly)	Registers dynamically compiled migrations
MigrationsOperations.AddAndApplyMigration()	Orchestrates scaffold → compile → register → apply workflow

Design Decisions

• Extends existing services: Uses IMigrationsScaffolder for scaffolding and IMigrator for applying, adding only the new IMigrationCompiler service
• AddMigrations(Assembly): Extended IMigrationsAssembly interface to accept additional assemblies containing runtime-compiled migrations
• Always persists to disk: Like AddMigration, files are always saved to enable source control and future recompilation
• No pending changes behavior: If no model changes are detected, applies any existing pending migrations without creating a new one
• Internal compiler API: IMigrationCompiler and CSharpMigrationCompiler are in the .Internal namespace as they require design work for public API
• Error handling with cleanup: If compilation or migration application fails, saved migration files are cleaned up to prevent orphans
• Thread safety: MigrationsAssembly uses locking to protect against race conditions when adding migrations concurrently

Workflow

User runs: dotnet ef database update InitialCreate --add
    │
    ▼
MigrationsOperations.AddAndApplyMigration()
    │
    ├─► Check for pending model changes
    │       └─► If none: apply existing migrations, return
    │
    ├─► IMigrationsScaffolder.ScaffoldMigration() - Generate code
    │
    ├─► try {
    │       ├─► IMigrationsScaffolder.Save() - Write files to disk
    │       ├─► IMigrationCompiler.CompileMigration() - Roslyn compile
    │       ├─► IMigrationsAssembly.AddMigrations() - Register migration
    │       └─► IMigrator.Migrate() - Apply to database
    │   } catch {
    │       └─► Clean up saved files on failure
    │   }
    │
    └─► Return migration files

Robustness Features

1. Exception handling with cleanup: AddAndApplyMigration wraps the save-compile-register-apply chain in try-catch, deleting saved files on failure to prevent orphans
2. Context disposal on validation failure: PrepareForMigration ensures the DbContext is disposed if validation or service building fails
3. Thread-safe migration registration: MigrationsAssembly uses locking to protect shared state (migrations dictionary, model snapshot, additional assemblies list)

Limitations

• Requires dynamic code generation (incompatible with NativeAOT) - marked with [RequiresDynamicCode]
• C# only (no VB.NET/F# support)

Test plan

• Unit tests for CSharpMigrationCompiler
• Unit tests for MigrationsOperations.AddAndApplyMigration
• Integration tests in RuntimeMigrationTestBase (SQLite and SQL Server implementations)
• Tests for validation (empty name, invalid characters)
• Tests for RemoveMigration with dynamically created migrations
• All existing EFCore.Design.Tests pass
• All existing EFCore.Relational.Tests pass

Fixes #37342

[thromel]

Note on SQL Server Integration Tests

The RuntimeMigrationSqlServerTest tests are marked with [SqlServerCondition(SqlServerCondition.IsNotAzureSql | SqlServerCondition.IsNotCI)] and are skipped in CI. This follows the same pattern used by MigrationsInfrastructureSqlServerTest.

Why these tests are skipped in CI:

• They require creating fresh databases dynamically for each test to properly test the migration flow from scratch
• The Helix CI environment has SQL Server available but with limited permissions configured for shared/pre-configured databases
• Tests that use SqlServerTestStore.CreateInitializedAsync with dynamic database names don't work in the CI SQL Server setup

Test coverage is still maintained:

• Core runtime migration logic is covered by unit tests in EFCore.Design.Tests (which run in CI)
• The SQL Server integration tests run locally for developers with SQL Server configured
• SQLite integration tests in EFCore.Sqlite.FunctionalTests also validate the end-to-end flow

This is consistent with how other complex migration infrastructure tests handle CI limitations.

[thromel]

Thank you for the thorough review @AndriySvyryd! I've addressed all your feedback:

1. SQL Server tests: Now using static database name "RuntimeMigrationTest" and removed EnsureDeleted() calls since CreateInitializedAsync already cleans the database.

2. CLI validation: Added validation that shows an error if -o or -n is used without --add.

3. Renamed to AddAndApplyMigration: Renamed CreateAndApplyMigration to AddAndApplyMigration and reordered parameters as suggested.

4. Extracted common validation: Created ValidateMigrationName() and ValidateMigrationNameNotContextName() helper methods that are shared by both AddMigration and AddAndApplyMigration.

5. Added EnsureMigrationsAssembly call: Now calling EnsureMigrationsAssembly in AddAndApplyMigration.

6. Removed unnecessary IDesignTimeModel registration: Confirmed there's no duplicate registration.

7. Merged IDynamicMigrationsAssembly into IMigrationsAssembly: Added AddMigrations(Assembly) method to IMigrationsAssembly and deleted IDynamicMigrationsAssembly, DynamicMigrationsAssembly, and their tests.

8. Replaced CompiledMigration with AddMigrations(Assembly): IMigrationCompiler.CompileMigration now returns Assembly directly, and RuntimeMigrationService uses _migrationsAssembly.AddMigrations() to register compiled migrations. Removed the CompiledMigration class.

All tests pass locally (SQLite: 26 tests, SQL Server: 7 tests, CSharpMigrationCompiler: 4 tests, MigrationsOperations: 2 tests).

[AndriySvyryd]

Create default values for OutputDirectory and Namespace if not specified, like in AddMigration above. In general, try combine this and AddMigration into a single method.

[AndriySvyryd]

Don't create a new service for this, distribute the functionality among the existing services like IMigrationsScaffolder and IMigrator,

[thromel]

Addressed the additional review feedback:

1. Moved validation to Validate() - DatabaseUpdateCommand now uses Validate() override instead of validating in Execute()

2. Removed extension methods - CreateAndApplyMigration and CreateAndApplyMigrationAsync removed from RelationalDatabaseFacadeExtensions

3. Removed PersistToDisk - Files are always persisted to disk (like AddMigration)

4. Using defaults like AddMigration - AddAndApplyMigration now uses SubnamespaceFromOutputPath() for namespace defaults

5. Distributed RuntimeMigrationService to existing services:

   • Deleted IRuntimeMigrationService, RuntimeMigrationOptions, RuntimeMigrationResult
   • Deleted RuntimeMigrationService
   • Moved functionality into MigrationsOperations.AddAndApplyMigration()
   • Uses existing IMigrationsScaffolder, IMigrationCompiler, IMigrationsAssembly, and IMigrator

6. No longer throws if no pending changes - Instead logs an info message and applies existing migrations

Files changed: 11 files, +132/-2064 lines (significant cleanup)

[thromel]

Question on NonCapturingLazyInitializer for Migrations Property

Per review comment, I switched Migrations from lock-based caching to NonCapturingLazyInitializer. However, after some analysis, there's a potential race condition:

The Issue

NonCapturingLazyInitializer uses Interlocked.CompareExchange which can race with AddMigrations() cache invalidation:

Thread A: Starts computing Migrations (sees _migrations == null)
Thread B: Calls AddMigrations(), adds assembly, sets _migrations = null
Thread A: Finishes computing (without new assembly), stores via CompareExchange
Result: Stale dictionary permanently cached, missing new assembly's migrations

Current Implementation

// Migrations - uses NonCapturingLazyInitializer (per review request)
public virtual IReadOnlyDictionary<string, TypeInfo> Migrations
    => NonCapturingLazyInitializer.EnsureInitialized(
        ref _migrations,
        this,
        static self => { /* compute */ });

// AddMigrations - invalidates cache
public virtual void AddMigrations(Assembly assembly)
{
    lock (_lock)
    {
        _additionalAssemblies.Add(assembly);
        _migrations = null;  // <-- Race: can be overwritten by concurrent EnsureInitialized
        _modelSnapshot = null;
        _modelSnapshotInitialized = false;
    }
}

Options

1. Keep NonCapturingLazyInitializer - Accept the race (unlikely in practice since AddMigrations is design-time only)
2. Revert to lock-based caching - Consistent with ModelSnapshot property, eliminates race
3. Versioned cache - More complex but allows lock-free reads

Practical Risk Assessment

The race is unlikely because:

• AddMigrations is only called during design-time database update --auto operations
• It's not expected to be called concurrently with Migrations access
• The scenario requires very specific timing

Please advise on preferred approach. Happy to adjust either way.