63 — Forgeyard Database Schema Migration, Online Backfill, Data Transformation & Zero-Downtime Change Orchestration System Architecture

Document type: Core Database Schema Migration, Online Backfill, Data Transformation, Expand-Contract, Cutover, Rollback & Zero-Downtime Change Orchestration System Architecture
Project: Forgeyard CI/CD
Subsystem: database schema evolution, migration plans, migration generations, DDL orchestration, online backfills, chunked data transformation, expand-contract rollout, dual-read/write transitions, compatibility windows, migration verification, cutover, rollback limits, tenant-by-tenant migration, drift detection, resumability, throttling, migration locks, and recovery
Architecture style: Explicit migration state, compatibility-first evolution, resumable data work, bounded batches, idempotent checkpoints, exact schema generations, separate schema/data phases, guarded cutovers, fail-safe recovery, and no opaque migrate up hidden in deployment scripts
Status: Target production architecture extending the Forgeyard series beyond the original 01–26 plan
Relationship to prior work: Builds on Storage/Metadata, Deployment, Infrastructure-as-Code, Compatibility Governance, Test Data, Concurrency/Fencing, Progressive Delivery, Reliability/SLOs, Multi-Tenancy, Audit, Incident Management, and Data Lifecycle. This subsystem provides the first-class state machine required for safe online schema and data evolution.

1. Purpose

Database changes are among the highest-risk effects in a production system.

They can involve:

DDL
indexes
constraints
column/type changes
table rewrites
backfills
data normalization
key migration
encryption changes
tenant moves
denormalization
partitioning

Naive CI/CD often treats migration as:

deploy app
  ↓
run `migrate up`
  ↓
hope it finishes

That is insufficient for:

large datasets
rolling deployments
multi-version clients
multi-region systems
high availability
tenant isolation
long-running backfills
rollback

The central rule is:

A database migration is a first-class durable workflow with explicit phases, checkpoints, compatibility windows, authority, verification, and recovery semantics.

A second rule is:

Schema migration and data migration are related but distinct. DDL completion does not imply data transformation completion.

A third rule is:

Rollback is never assumed. Every migration declares whether rollback is safe, partial, manual, or impossible.

2. Architectural Position

                   Proposed Schema/Data Change
                              │
                              ▼
                       Migration Plan
                              │
                              ▼
                     Compatibility Check
                              │
                              ▼
                          Expand Phase
                              │
                              ▼
                      Application Rollout
                              │
                              ▼
                         Backfill Phase
                              │
                              ▼
                        Verify / Cutover
                              │
                              ▼
                         Contract Phase

3. Goals

The subsystem MUST:

define migration identity;
define schema generation identity;
define migration plan identity;
support expand-contract;
support online DDL;
support resumable backfills;
support chunking;
support throttling;
support checkpoints;
support dual-read/write transitions;
support cutover;
support verification;
support tenant-by-tenant migration;
support multi-region migration authority;
support zero/low-downtime migration;
support rollback classification;
support manual recovery;
support migration locks/fencing;
support schema drift detection;
support migration test environments;
support dry-run/planning;
support observability;
support Dioxus UI/API/CLI;
support audit;
support incident integration;
support DR;
support air-gap;
support large datasets;
support policy gates;
prevent opaque migration side effects hidden in deployments.

4. Non-Goals

This subsystem does not replace:

database engines
ORM migration tools
SQL parsers
application compatibility testing
backup systems
deployment orchestration

It governs them.

5. Workspace Structure

crates/migration/
├── forgeyard-migration/
├── forgeyard-migration-model/
├── forgeyard-migration-plan/
├── forgeyard-migration-schema/
├── forgeyard-migration-backfill/
├── forgeyard-migration-cutover/
├── forgeyard-migration-verify/
├── forgeyard-migration-drift/
├── forgeyard-migration-reconcile/
├── forgeyard-migration-health/
└── forgeyard-migration-testkit/

Adapters:

crates/migration-adapters/
├── forgeyard-migration-postgres/
├── forgeyard-migration-stoolap/
├── forgeyard-migration-sqlite/
└── forgeyard-migration-custom/

6. MigrationId

#![allow(unused)]
fn main() {
pub struct MigrationId(Ulid);
}

One durable migration workflow.

7. MigrationPlanId

#![allow(unused)]
fn main() {
pub struct MigrationPlanId(Digest);
}

Immutable identity of exact migration intent.

8. SchemaGenerationId

#![allow(unused)]
fn main() {
pub struct SchemaGenerationId(Digest);
}

Represents exact logical schema generation.

9. DatabaseTarget

#![allow(unused)]
fn main() {
pub struct DatabaseTarget {
    pub database: DatabaseId,
    pub tenant: Option<TenantId>,
    pub region: Option<RegionId>,
}
}

10. MigrationPlan

#![allow(unused)]
fn main() {
pub struct MigrationPlan {
    pub id: MigrationPlanId,
    pub target: DatabaseTarget,
    pub from: SchemaGenerationId,
    pub to: SchemaGenerationId,
    pub phases: Vec<MigrationPhaseSpec>,
    pub compatibility: CompatibilityReportId,
}
}

11. Migration Phase

#![allow(unused)]
fn main() {
pub enum MigrationPhaseSpec {
    Expand(SchemaChangeSet),
    DeployCompatibleApplication(ReleaseId),
    Backfill(BackfillSpec),
    Verify(VerificationSpec),
    Cutover(CutoverSpec),
    Contract(SchemaChangeSet),
}
}

12. Expand-Contract Baseline

Preferred online sequence:

old schema
   ↓
expand with backward-compatible changes
   ↓
old + new app versions coexist
   ↓
backfill
   ↓
switch reads/writes
   ↓
verify
   ↓
contract old fields

13. MigrationState

#![allow(unused)]
fn main() {
pub enum MigrationState {
    Planned,
    AwaitingApproval,
    Expanding,
    WaitingForApplication,
    Backfilling,
    Verifying,
    AwaitingCutover,
    CuttingOver,
    Contracting,
    Succeeded,
    Paused,
    Failed,
    Cancelled,
    Unknown,
}
}

14. Unknown

First-class for ambiguous external/database effects.

Never treated as success.

15. Schema Change Classification

#![allow(unused)]
fn main() {
pub enum SchemaChangeRisk {
    Additive,
    OnlineCompatible,
    RequiresRewrite,
    Destructive,
    Irreversible,
    Unknown,
}
}

16. Additive Change

Examples:

new nullable column
new table
new non-blocking index where supported

17. Potentially Blocking Change

Examples:

large table rewrite
type conversion
constraint validation
exclusive lock requirement

#![allow(unused)]
fn main() {
pub struct MigrationEngineCapabilities {
    pub concurrent_index: bool,
    pub online_constraint_validation: bool,
    pub transactional_ddl: bool,
    pub lock_timeout_control: bool,
}
}

21. Migration Source

Can come from:

SQL file
ORM-generated migration
Forgeyard-native typed migration spec
custom engine adapter

#![allow(unused)]
fn main() {
pub struct SchemaSnapshot {
    pub generation: SchemaGenerationId,
    pub observed_at: Timestamp,
    pub engine_version: DatabaseEngineVersion,
}
}

25. Desired vs Observed

Critical distinction.

26. Schema Drift

#![allow(unused)]
fn main() {
pub enum SchemaDrift {
    InSync,
    UnexpectedObject,
    MissingObject,
    ModifiedObject,
    Unknown,
}
}

27. Migration Preconditions

Before start:

current schema generation matches expected
no incompatible active migration
backup/recovery requirements satisfied
compatibility report valid
required application versions available

28. Schema Generation Precondition

No blind migration from unknown schema.

29. Migration Lock

Part 60.

Scope:

DatabaseId / tenant shard / schema domain

expected schema
statement digest
timeout
lock budget
retry safety

33. StatementId

#![allow(unused)]
fn main() {
pub struct MigrationStatementId(Digest);
}

add constraint not validated
  ↓
backfill/fix data
  ↓
validate

43. Column Rename

Safer pattern:

add new column
dual-write
backfill
switch reads
stop old writes
drop old later

#![allow(unused)]
fn main() {
pub struct BackfillId(Ulid);
}

48. BackfillSpec

#![allow(unused)]
fn main() {
pub struct BackfillSpec {
    pub source: BackfillSource,
    pub target: BackfillTarget,
    pub chunking: ChunkingStrategy,
    pub throttle: BackfillThrottle,
    pub verification: BackfillVerification,
}
}

49. Chunking Strategy

#![allow(unused)]
fn main() {
pub enum ChunkingStrategy {
    PrimaryKeyRange,
    TimestampRange,
    HashBuckets,
    Partition,
    Custom(ChunkingStrategyId),
}
}

50. ChunkId

#![allow(unused)]
fn main() {
pub struct BackfillChunkId(Digest);
}

51. Chunk State

#![allow(unused)]
fn main() {
pub enum BackfillChunkState {
    Pending,
    Running,
    Succeeded,
    Failed,
    Retryable,
    Skipped,
}
}

#![allow(unused)]
fn main() {
pub struct BackfillCheckpoint {
    pub backfill: BackfillId,
    pub completed: u64,
    pub last_position: Option<BackfillCursor>,
}
}

55. No “start from row 1 again”

Critical for large datasets.

56. Backfill Throttle

#![allow(unused)]
fn main() {
pub struct BackfillThrottle {
    pub rows_per_second: Option<u64>,
    pub max_concurrency: u16,
    pub max_db_cpu_percent: Option<Percentage>,
}
}

57. Adaptive Throttling

Can react to:

DB CPU
replication lag
lock wait
query latency
error rate

row counts
checksums
sample verification
full predicate verification
application-level invariant

65. Verification Confidence

#![allow(unused)]
fn main() {
pub enum BackfillVerificationConfidence {
    Full,
    Strong,
    Sampled,
    Weak,
    Unknown,
}
}

#![allow(unused)]
fn main() {
pub enum DualWriteState {
    Disabled,
    Shadow,
    Enabled,
    NewPrimary,
    OldDisabled,
}
}

#![allow(unused)]
fn main() {
pub enum ReadMode {
    OldOnly,
    OldPrimaryCompareNew,
    NewPrimaryFallbackOld,
    NewOnly,
}
}

73. Data Divergence

First-class finding.

74. DivergenceFindingId

#![allow(unused)]
fn main() {
pub struct DivergenceFindingId(Ulid);
}

75. Cutover

Switch authority from old representation to new.

76. CutoverId

#![allow(unused)]
fn main() {
pub struct CutoverId(Ulid);
}

77. Cutover Preconditions

backfill complete
verification passed
application compatibility valid
replication healthy
incident/change-freeze policy permits
rollback class understood

78. Cutover Is High-Risk

Often manual approval required.

79. Cutover State

#![allow(unused)]
fn main() {
pub enum CutoverState {
    Planned,
    AwaitingApproval,
    Executing,
    Verified,
    Failed,
    Unknown,
}
}

#![allow(unused)]
fn main() {
pub enum MigrationRollbackClass {
    FullyReversible,
    ReversibleBeforeCutover,
    ReversibleWithDataLossRisk,
    RollForwardOnly,
    ManualRecovery,
    Irreversible,
    Unknown,
}
}

87. Rollback Plan

#![allow(unused)]
fn main() {
pub struct MigrationRollbackPlan {
    pub class: MigrationRollbackClass,
    pub steps: Vec<RollbackStep>,
}
}

Expand schema
  ↓
deploy app version supporting old+new
  ↓
backfill
  ↓
switch reads

old schema + old app
expanded schema + old app
expanded schema + new app
backfilled schema + new app

plan SQL
estimate locks
estimate rows
estimate rewrite

#![allow(unused)]
fn main() {
pub enum MigrationRisk {
    Low,
    Moderate,
    High,
    Critical,
    Unknown,
}
}

107. Risk Inputs

DDL lock class
table size
row rewrite
backfill volume
irreversibility
production criticality
multi-region scope

108. Policy Gates

Can require:

backup evidence
maintenance window
manual approval
compatibility report
mixed-version tests
rollback plan
capacity headroom

#![allow(unused)]
fn main() {
pub struct TenantMigrationId(Ulid);
}

115. Tenant Rollout

pilot tenant
  ↓
small cohort
  ↓
larger cohort
  ↓
all tenants

#![allow(unused)]
fn main() {
pub struct ShardMigrationId(Ulid);
}

#![allow(unused)]
fn main() {
pub enum DowntimeRequirement {
    NoneExpected,
    BriefMaintenance,
    PlannedMaintenance,
    Unknown,
}
}

freeze migration
require adoption
reconcile

140. Brownfield Adoption

Existing DB schema can be imported as generation baseline.

141. Adoption Requires

introspection
owner approval
no hidden pending migration
backup/recovery state

142. Migration Artifact Signing

Optional high assurance.

143. Migration Provenance

#![allow(unused)]
fn main() {
pub struct MigrationProvenance {
    pub plan: MigrationPlanId,
    pub source: SourceSnapshotId,
    pub release: Option<ReleaseId>,
    pub compatibility: CompatibilityReportId,
}
}

144. Audit

Audit:

migration approval
cutover
rollback attempt
force resume
manual checkpoint override
contract phase
drift adoption

145. Routine Chunk Completion

Operational telemetry, not privileged audit spam.

146. Dioxus UI

Pages:

Database Migrations
Migration Plans
Backfills
Schema Drift
Cutovers
Migration History

147. Migration Detail

Shows:

from/to schema
phase
progress
risk
compatibility
backfill
cutover
rollback class

148. Backfill Progress

Shows:

rows/chunks complete
throughput
ETA estimate
DB health
pause reason

149. ETA

Estimate only.

150. CLI

forgeyard migrate plan
forgeyard migrate apply
forgeyard migrate status
forgeyard migrate pause
forgeyard migrate resume
forgeyard migrate verify
forgeyard migrate cutover
forgeyard migrate rollback
forgeyard migrate drift
forgeyard migrate doctor

151. API

Potential:

POST /v1/migrations
GET  /v1/migrations/{id}
POST /v1/migrations/{id}/start
POST /v1/migrations/{id}/pause
POST /v1/migrations/{id}/resume
POST /v1/migrations/{id}/cutover
POST /v1/migrations/{id}/rollback

152. Permissions

migration.read
migration.plan
migration.apply
migration.pause
migration.resume
migration.cutover
migration.rollback
migration.force

#![allow(unused)]
fn main() {
pub struct TransformerId(Digest);
}

dual-write
change-data-capture
version predicate
reconciliation pass

migration_duration
backfill_rate
chunk_failure_rate
DB CPU
replication lag
lock wait
cutover latency

181. SLO Guard

Can pause if production health degrades.

182. Incident Integration

Part 61.

Migration causing incident:

pause
link IncidentId
freeze further cutover

183. Incident Resolution

Does not auto-resume migration.

184. Resume Freshness

Re-check:

schema state
compatibility
backfill
health
policy

185. Progressive Delivery

Part 62.

Migration phases can gate application promotion.

186. Example

schema expand complete
  ↓
allow app canary

#![allow(unused)]
fn main() {
pub struct LocalMigrationPlanId(Digest);
}

client schema version
server schema version
sync protocol version

must be compatible.

destructive migration abuse
SQL injection in migration tooling
privilege escalation
unbounded backfill
data corruption
schema drift
stale cutover controller

migrations_active
migration_phase_duration_seconds
backfill_chunks_total
backfill_chunks_failed_total
backfill_rows_processed_total
migration_paused_total
migration_cutover_unknown_total
schema_drift_findings_total

217. Tracing

migration.plan
migration.expand
migration.backfill
migration.verify
migration.cutover
migration.contract
migration.reconcile

218. Health

#![allow(unused)]
fn main() {
pub enum MigrationSubsystemHealth {
    Healthy,
    BackfillDegraded,
    DriftDetected,
    CutoverBlocked,
    Unhealthy,
}
}

219. Doctor

forgeyard migrate doctor

Checks:

stale migration lock
schema drift
stuck backfill
repeated poison chunks
unknown cutover
missing rollback plan
old app still active before contract

220. Search

Part 31 indexes migration metadata/history.

221. Cost

Part 45.

Backfill consumes:

DB compute
IO
replication bandwidth
storage

restore DB
  ↓
detect schema generation
  ↓
load migration metadata
  ↓
invalidate stale leases
  ↓
reconcile

#![allow(unused)]
fn main() {
pub struct MigrationBundle {
    pub plan: MigrationPlanId,
    pub statements: CasObjectRef,
    pub transformer: Option<TransformerId>,
    pub verification: CasObjectRef,
}
}

232. Signed Bundle

Optional high assurance.

233. Testkit

forgeyard-migration-testkit/src/
├── lib.rs
├── schema.rs
├── plan.rs
├── ddl.rs
├── backfill.rs
├── cutover.rs
├── rollback.rs
├── drift.rs
└── assertions.rs

234. Core Tests

Schema

expected generation required;
drift blocks unsafe start;
additive change classified correctly.

Expand-Contract

old app works after expand;
new app works against transitional schema;
contract blocked while old app exists.

Backfill

chunk checkpoints durable;
retries idempotent;
poison chunk visible;
throttling obeys DB health.

Dual-Write

divergence detected;
no lost tail writes.

Cutover

prerequisites enforced;
timeout becomes Unknown;
stale controller fenced.

Rollback

irreversible migration never gets fake rollback;
rollback target compatibility checked.

Tenant/Shards

pilot cohort;
one tenant failure isolated;
shard rollout respects readiness.

DR/Federation

restored state reconciles;
authority failover does not duplicate cutover.

235. Chaos Tests

Inject:

DB failover during DDL
worker crash during backfill
network partition
replica lag spike
cutover API timeout
controller restart
storage pressure

Expected behavior:

checkpoint survives
no duplicate destructive action
migration pauses/reconciles
health remains explicit

236. Scale Tests

Test:

billions of rows via chunk simulation
thousands of tenant migrations
hundreds of shards
long-running multi-day backfills

Migration binds exact source and target schema generations.
Migration plan is immutable/versioned.
Schema and data migration phases remain distinct.
Expand-contract is baseline for rolling-compatible changes.
Current observed schema must match expected baseline.
Schema drift cannot be ignored silently.
Migration lock and schema-generation check are both required where applicable.
Stale migration controller is fenced.
DDL retry safety is engine-specific.
Generic blind DDL retry is forbidden.
Long-running backfills are resumable.
Backfill chunks have durable checkpoints.
Chunk transformation version is immutable for a migration.
Backfill can throttle based on production DB health.
Missing verification does not permit cutover.
Unknown cutover state does not become success.
Rollback class is explicit.
Irreversible migration never exposes fake rollback.
Old application versions remain supported through transitional schema as policy requires.
Contract phase is blocked until old clients/apps are no longer supported.
Tenant/shard migration failures are isolated.
Multi-region authority prevents duplicate schema mutation.
Production-derived migration test data obeys Part 56 controls.
Migration worker uses least-privilege credentials.
Build jobs do not receive production DDL credentials.
Active incident/change freeze can pause migration.
Resume re-checks schema/health/policy freshness.
DR reconciles restored DB schema against migration metadata.
Local embedded DB migration supports crash recovery.
Forgeyard dogfoods the subsystem for its own Postgres/Stoolap schema evolution.

239. Production Readiness Gates

Do not call migration orchestration production-ready until:

schema generation identity is stable
Postgres expand-contract path is dogfooded
backfill checkpoints survive crashes
cutover Unknown reconciliation is proven
rollback classification is enforced
schema drift detection works
tenant/shard isolation passes
migration credentials are least privilege
DR/federation reconciliation passes
chaos/scale tests pass

240. Architectural Invariants

migration is first-class durable state;
schema generation is explicit;
migration plan is immutable;
schema and data phases are distinct;
expand-contract is preferred;
current schema must match expected baseline;
drift is first-class;
migration ownership is fenced;
DDL retries are operation-specific;
backfills are resumable;
chunk progress is durable;
transformation logic cannot change silently mid-run;
verification precedes cutover;
Unknown is not success;
rollback capability is explicit/honest;
contract waits for compatibility window;
old/new apps coexist where rolling upgrade requires;
migration workers use scoped credentials;
production DDL authority is not given to normal build jobs;
reliability signals can throttle/pause data work;
incidents/change freezes can pause migration;
tenant/shard failures remain scoped;
federation authority prevents duplicate mutation;
DR reconciles actual schema before continuation;
local DB migration is crash-safe;
data lifecycle/residency apply to transformed data;
audit captures cutover/override/destructive actions;
no fake zero-downtime guarantee;
no opaque migration side effects hidden in deployment scripts;
Forgeyard dogfoods its own migration system.

241. Final Target Architecture

                 MigrationPlanId
                       │
                       ▼
                  Schema Expand
                       │
                       ▼
              Compatible App Rollout
                       │
                       ▼
                Resumable Backfill
                       │
                       ▼
                    Verify
                       │
                       ▼
                    Cutover
                       │
                       ▼
               Compatibility Window
                       │
                       ▼
                    Contract

Backfill:

large dataset
    ↓
deterministic chunking
    ↓
lease chunk
    ↓
transform
    ↓
verify
    ↓
checkpoint
    ↓
resume until complete

Cutover:

backfill complete
+
verification passed
+
app compatibility valid
+
health acceptable
+
policy approval
   ↓
cutover
   ↓
observe actual state
   ↓
verified success / Unknown / recovery

The key guarantee is:

Forgeyard treats schema and data evolution as a governed distributed workflow rather than a one-shot script. Every migration has exact generations, compatibility evidence, resumable checkpoints, health-aware throttling, explicit cutover authority, and honest rollback semantics—so large online changes remain observable and recoverable even across crashes, failovers, and long-running transformations.

242. Extended Architecture Sequence

01 Core Domain & Foundation
02 Storage & Metadata
03 CAS & Artifact Data Plane
04 Pipeline IR / Parsing / Planning
05 Run / Job State Machine
06 Scheduler
07 Runner / Agent
08 Sandbox & Executor
09 Transport / QUIC / Protocol
10 Events / Reconciliation
11 Policy / Authorization / Identity
12 Secrets / Trust
13 Supply Chain / SBOM / Provenance / Signing
14 Packaging
15 Release
16 Deployment
17 Observability / Health / Doctor
18 API / Axum
19 Dioxus UI
20 Device Lab
21 SCM Provider Integrations
22 HA / Coordination / Raft
23 RBE Interop
24 Plugin / Extension Architecture
25 Operations / Backup / Upgrade / DR
26 Self-Hosting / Bootstrap / Release Forgeyard
27 Multi-Tenancy / Quotas / Resource Governance
28 Audit / Compliance / Security Governance
29 Notifications / Alerting / Human Workflow
30 Entitlements / Licensing / Subscription
31 Search / Indexing / Operational Analytics
32 Test Results / Quality / Coverage / Flaky Intelligence
33 Benchmark / Performance / Load / Capacity
34 Monorepo / Dependency Graph / Affected Work
35 Developer Experience / Local Dev
36 Dependency / Registry / Mirror Governance
37 Static Analysis / Security Findings
38 Cache / Remote Cache / Correctness
39 Configuration / Feature Flags / Runtime
40 Security / Threat Model / Incident Response
41 Release Distribution / Update Delivery
42 Workflow Templates / Golden Paths
43 Runner Fleet Autoscaling / Provisioning
44 Pipeline Triggers / Schedules / Dispatch
45 Cost Accounting / FinOps
46 Data Lifecycle / Retention / Privacy
47 CI/CD Migration / Compatibility
48 Failure Diagnosis / Reproduction / Bisect
49 Service Catalog / Ownership / Developer Portal
50 Reliability / SLO / Error Budget / Resilience
51 Multi-Region Federation / Edge / Disconnected
52 Artifact Registry / OCI / Package Distribution
53 Infrastructure-as-Code / Preview Environments / Drift
54 Merge Queue / Speculative Integration / Batch Validation
55 AI-Assisted CI Optimization / Engineering Copilot
56 Test Data / Fixtures / Ephemeral Databases / Service Virtualization
57 API / ABI / Schema / Protocol Compatibility / Contract Evolution
58 Runner Image Factory / Golden Images / Patch Management / Baseline Attestation
59 Network Connectivity / Private Resource Access / Egress / Tunneling / Zero-Trust Connectivity
60 Workflow Concurrency / Distributed Locks / Idempotency / Reservations / Exclusive Coordination
61 Incident Management / On-Call / Escalation / Response Coordination / Postmortem
62 Environment Promotion / Progressive Delivery / Feature Rollout / Canary Analysis / Automated Rollback
63 Database Schema Migration / Online Backfill / Data Transformation / Zero-Downtime Change Orchestration

Keyboard shortcuts

Forgeyard System Architecture & Engineering Documentation