65 — Forgeyard Build Graph Replay, Historical Reproducibility, Time-Travel CI & Evidence Reconstruction System Architecture

Document type: Core Historical Build Replay, Time-Travel CI, Evidence Reconstruction, Archived Dependency Resolution & Reproducibility Recovery System Architecture
Project: Forgeyard CI/CD
Subsystem: historical run reconstruction, source/toolchain/config snapshot replay, archived dependency resolution, historical pipeline planning, exact environment reconstruction, old runner/image compatibility, evidence-gap handling, deterministic replay, semantic replay, best-effort reconstruction, historical debug, compliance replay, incident replay, and long-term reproducibility governance
Architecture style: Immutable historical identities, replay manifests, evidence-first reconstruction, explicit fidelity levels, archived inputs, compatibility-aware execution, provenance preservation, no silent substitution, and honest handling of irrecoverable historical state
Status: Target production architecture extending the Forgeyard series beyond the original 01–26 plan
Relationship to prior work: Builds on Core Domain, CAS, Pipeline IR, Reproducibility/Hermetic Packaging, Toolchains, Runner Images, Test Data, Dependency Mirrors, Compatibility Governance, Failure Diagnosis, Audit, Data Lifecycle, Release Provenance, Self-Hosting, and Incident/Postmortem systems. This subsystem enables exact or bounded reconstruction of historical execution.

1. Purpose

Over time, engineers need to answer questions such as:

Can we reproduce run #123 from six months ago?
Why did this artifact differ from today’s rebuild?
Which toolchain and dependency versions were used?
Can we recreate the exact integration candidate?
Can we replay the same failing test?
Can an auditor verify how this release was produced?
Can we rerun a historical pipeline after an incident?

Naive CI systems often retain only:

logs
commit SHA
status

That is insufficient.

A reliable replay needs some or all of:

exact source snapshot
pipeline IR
toolchain descriptors
dependency lockfiles
resolved package digests
configuration snapshot
policy digest
test-data identity
runner baseline
environment identity
network policy
secrets references
external service observations

The central rule is:

Historical replay is possible only to the fidelity supported by retained immutable inputs and external-state evidence. Forgeyard never claims exact replay when required historical inputs are missing or mutable external systems cannot be reconstructed.

A second rule is:

Replay never substitutes newer dependencies, newer toolchains, newer base images, or newer configuration silently. Every substitution is explicit and lowers replay fidelity.

A third rule is:

Historical replay creates a new execution record linked to the original; it never rewrites the historical Run/Job/evidence that actually occurred.

2. Architectural Position

                    Historical Run
                         │
                         ▼
                   Replay Manifest
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
            Source    Toolchains   Config
              │          │          │
              └──────────┼──────────┘
                         ▼
                Archived Dependencies
                         │
                         ▼
                Replay Environment Plan
                         │
                         ▼
                     New Replay Run
                         │
                         ▼
                 Compare / Reconstruct

3. Goals

The subsystem MUST:

define replay identity;
define replay manifest identity;
reconstruct exact source snapshot;
reconstruct pipeline IR;
reconstruct exact toolchains;
reconstruct dependency resolution;
reconstruct relevant config;
reconstruct test data/environment where possible;
reconstruct runner baseline where possible;
support deterministic replay;
support semantic replay;
support best-effort replay;
support historical failure reproduction;
support release verification replay;
support incident replay;
support compliance/audit replay;
support evidence reconstruction;
support old-version executors/tooling;
support replay compatibility bridges;
support replay bundles;
support air-gap replay;
support missing-input detection;
support explicit substitution;
support fidelity scoring/classification;
support UI/API/CLI;
support retention policy;
support audit;
support security;
support federation/DR;
never mutate historical truth.

4. Non-Goals

This subsystem does not:

guarantee impossible reconstruction of deleted external state
guarantee bit-for-bit output for inherently nondeterministic software
replace normal CI
replace artifact provenance
replace backup
replace archival policy

5. Workspace Structure

crates/replay/
├── forgeyard-replay/
├── forgeyard-replay-model/
├── forgeyard-replay-manifest/
├── forgeyard-replay-source/
├── forgeyard-replay-toolchain/
├── forgeyard-replay-dependency/
├── forgeyard-replay-environment/
├── forgeyard-replay-evidence/
├── forgeyard-replay-compare/
├── forgeyard-replay-reconcile/
├── forgeyard-replay-health/
└── forgeyard-replay-testkit/

6. ReplayId

#![allow(unused)]
fn main() {
pub struct ReplayId(Ulid);
}

One new execution attempt intended to reconstruct an earlier execution.

7. ReplayManifestId

#![allow(unused)]
fn main() {
pub struct ReplayManifestId(Digest);
}

Immutable identity of historical replay inputs.

8. Replay Manifest

#![allow(unused)]
fn main() {
pub struct ReplayManifest {
    pub id: ReplayManifestId,
    pub original_run: RunId,
    pub source: SourceSnapshotId,
    pub pipeline: PipelineIrId,
    pub toolchains: Vec<ToolchainDescriptorId>,
    pub dependencies: Vec<ResolvedDependencySetId>,
    pub config: ConfigSnapshotId,
    pub policy: PolicyDigest,
    pub test_data: Vec<TestDatasetId>,
    pub runner_baseline: Option<RunnerBaselineId>,
}
}

9. Manifest Source

Prefer generated automatically at original run time.

10. No Forensic Guessing First

Critical.

If exact manifest exists, use it.

For old runs predating manifest support, use reconstruction mode.

11. Replay Fidelity

#![allow(unused)]
fn main() {
pub enum ReplayFidelity {
    BitForBitExpected,
    DeterministicInputs,
    EquivalentEnvironment,
    SemanticReplay,
    BestEffort,
    EvidenceOnly,
    Impossible,
}
}

#![allow(unused)]
fn main() {
pub enum ReplaySubject {
    Run(RunId),
    Job(JobId),
    JobAttempt(JobAttemptId),
    Release(ReleaseId),
    IntegrationCandidate(IntegrationCandidateId),
    FailureObservation(FailureObservationId),
    Incident(IncidentId),
}
}

21. ReplayIntent

#![allow(unused)]
fn main() {
pub enum ReplayIntent {
    Debug,
    VerifyReproducibility,
    Audit,
    IncidentAnalysis,
    SecurityAnalysis,
    Regression,
    Compliance,
}
}

#![allow(unused)]
fn main() {
pub struct PipelineIrVersion(u16);
}

#![allow(unused)]
fn main() {
pub struct ReplayCompatibilityBridgeId(Digest);
}

Transforms historical IR/schema into executable current internal representation while preserving semantics where proven.

compiler
linker
SDK
runtime
system headers
package managers

37. Toolchain Availability

#![allow(unused)]
fn main() {
pub enum HistoricalArtifactAvailability {
    Available,
    Rebuildable,
    SubstituteRequired,
    Missing,
}
}

#![allow(unused)]
fn main() {
pub struct ResolvedDependencySetId(Digest);
}

exact image artifact
provider image copy
rebuild from image definition
equivalent baseline
unavailable

51. Exact Host Hardware

Often unavailable.

52. HardwareProfile

#![allow(unused)]
fn main() {
pub struct HardwareProfile {
    pub arch: Architecture,
    pub cpu_features: CpuFeatureSet,
    pub gpu: Option<GpuIdentity>,
    pub memory_class: MemoryClass,
}
}

FixtureSetId
TestDatasetId
TestEnvironmentSpecId
random seed

#![allow(unused)]
fn main() {
pub enum ReplaySecretMode {
    Deny,
    CurrentEquivalent,
    HistoricalEscrow,
    Synthetic,
}
}

#![allow(unused)]
fn main() {
pub enum ReplayNetworkMode {
    ExactOfflineInputs,
    Virtualized,
    CurrentExternal,
    Denied,
}
}

#![allow(unused)]
fn main() {
pub struct ReplayBundleId(Digest);
}

77. Replay Bundle Contents

Potential:

ReplayManifest
source snapshot
pipeline IR
toolchain closure
dependency closure
test fixtures
config snapshots
non-secret environment metadata
required protocol schemas

#![allow(unused)]
fn main() {
pub struct ReplayPlanId(Digest);
}

82. Replay Plan

#![allow(unused)]
fn main() {
pub struct ReplayPlan {
    pub id: ReplayPlanId,
    pub manifest: ReplayManifestId,
    pub fidelity: ReplayFidelity,
    pub substitutions: Vec<ReplaySubstitution>,
}
}

83. ReplaySubstitution

#![allow(unused)]
fn main() {
pub struct ReplaySubstitution {
    pub original: HistoricalInputRef,
    pub replacement: HistoricalInputRef,
    pub reason: ReplaySubstitutionReason,
}
}

84. Substitution Reasons

#![allow(unused)]
fn main() {
pub enum ReplaySubstitutionReason {
    MissingArtifact,
    SecurityRevocation,
    UnsupportedRuntime,
    PrivacyRestriction,
    UserRequested,
}
}

#![allow(unused)]
fn main() {
pub enum ForensicReplayClass {
    Normal,
    Restricted,
    Quarantined,
    Forbidden,
}
}

#![allow(unused)]
fn main() {
pub enum ReplayPolicyMode {
    OriginalForAnalysis,
    CurrentForExecutionSafety,
}
}

ReleaseId
  ↓
original derivation closure
  ↓
independent rebuild
  ↓
compare output digest

116. ReproductionResult

#![allow(unused)]
fn main() {
pub enum ReproductionResult {
    BitForBitMatch,
    NormalizedMatch,
    SemanticMatch,
    Mismatch,
    Inconclusive,
}
}

does failure still reproduce?
does exact environment reproduce?
does current environment reproduce?

123. Distinguish

ExactHistoricalReplay
CurrentEnvironmentReplay

124. Regression Diagnosis

Useful.

125. Incident Replay

Part 61.

Can reconstruct sequence of:

deploy
config
runner state
failure

production deploy
release publish
external webhook
payment-like effect

130. Replay Side-Effect Policy

#![allow(unused)]
fn main() {
pub enum ReplaySideEffectMode {
    Suppress,
    Virtualize,
    Sandbox,
    ExplicitLive,
}
}

#![allow(unused)]
fn main() {
pub enum ReplayClock {
    HistoricalFixed(Timestamp),
    Current,
    Virtual(TimeSimulationRef),
}
}

#![allow(unused)]
fn main() {
pub struct ReplayComparison {
    pub original: RunId,
    pub replay: RunId,
    pub output: Vec<OutputComparison>,
    pub evidence: Vec<EvidenceComparison>,
}
}

148. Output Comparison Types

#![allow(unused)]
fn main() {
pub enum OutputComparisonKind {
    Digest,
    NormalizedTree,
    Semantic,
    TestOutcome,
    Metric,
}
}

#![allow(unused)]
fn main() {
pub struct ReplayGap {
    pub input: HistoricalInputRef,
    pub reason: ReplayGapReason,
}
}

154. ReplayGapReason

#![allow(unused)]
fn main() {
pub enum ReplayGapReason {
    RetentionExpired,
    ExternalStateUnavailable,
    SecretUnavailable,
    ArtifactRevoked,
    UnsupportedTool,
    CorruptArchive,
    PrivacyRestriction,
    Unknown,
}
}

replay.read
replay.execute
replay.forensic
replay.export
replay.live_effect

forensic replay
historical secret escrow access
replay bundle export
live external side-effect enablement
security-revoked artifact execution

metadata only
evidence
source + pipeline
full reproducibility closure

171. ReplayRetentionClass

#![allow(unused)]
fn main() {
pub enum ReplayRetentionClass {
    Metadata,
    Evidence,
    ReproducibleInputs,
    FullClosure,
}
}

release
project criticality
compliance
incident

Historical Replay
Replay Plan
Replay Comparison
Replay Gaps
Replay Bundles

184. Run Detail

Button:

Replay

185. Replay Preview

Shows:

fidelity
available inputs
missing inputs
substitutions
security restrictions
estimated cost

186. Compare View

Side-by-side:

original
replay
outputs
tests
logs
environment

187. CLI

forgeyard replay plan <run>
forgeyard replay run <run>
forgeyard replay compare <replay>
forgeyard replay gaps <run>
forgeyard replay bundle create
forgeyard replay bundle inspect
forgeyard replay doctor

188. API

Potential:

POST /v1/replays/plan
POST /v1/replays
GET  /v1/replays/{id}
GET  /v1/replays/{id}/comparison
GET  /v1/runs/{id}/replay-availability

189. Replay Availability

#![allow(unused)]
fn main() {
pub struct ReplayAvailability {
    pub fidelity: ReplayFidelity,
    pub missing: Vec<ReplayGap>,
}
}

190. Doctor

forgeyard replay doctor

Checks:

missing release closures
corrupt archived toolchains
unreadable old PipelineIr
missing compatibility bridge
expired replay retention
replay bundle verification

191. Health

#![allow(unused)]
fn main() {
pub enum ReplaySubsystemHealth {
    Healthy,
    ArchiveDegraded,
    CompatibilityDegraded,
    ToolchainArchiveDegraded,
    Unhealthy,
}
}

192. Observability Metrics

replay_total
replay_exact_total
replay_best_effort_total
replay_failures_total
replay_gaps_total
replay_reproduction_mismatch_total

193. Labels

Low cardinality:

fidelity
intent
result

194. No RunId Labels

195. Tracing

replay.plan
replay.resolve
replay.materialize
replay.execute
replay.compare
replay.bundle

196. Compatibility

Part 57 central.

Historical replay needs compatibility for:

PipelineIr
protocol schemas
toolchain metadata
runner image formats
bundle formats

hardware
load
runner baseline
cache state

Often lower confidence.

executing vulnerable old toolchains
secret resurrection
malware replay
cross-tenant archive leakage
replaying destructive effects
tampered archive

225. Controls

current authz
current security floor
forensic isolation
no live effects by default
bundle signature verification
tenant scoping
no secret values in manifest

REPLAY / NON-PRODUCTION

where user-facing.

#![allow(unused)]
fn main() {
pub enum ReplayEvent {
    Planned,
    Materializing,
    Running,
    Completed,
    Failed,
    GapDetected,
    ComparisonCompleted,
}
}

234. Reconciler

Checks:

missing archive materialization
stuck replay
provider VM state
bundle import state
comparison completion

forgeyard-replay-testkit/src/
├── lib.rs
├── manifest.rs
├── archive.rs
├── environment.rs
├── replay.rs
├── compare.rs
├── gaps.rs
└── assertions.rs

239. Core Tests

Manifest

exact original identities captured;
manifest digest deterministic.

Source

deleted VCS branch still replayable if SourceSnapshot retained;
current branch never substituted automatically.

Toolchain

exact version resolved;
missing compiler produces gap;
substitute lowers fidelity.

Pipeline

archived IR preferred;
incompatible historical IR requires bridge.

Dependencies

upstream package deletion does not matter if mirrored;
missing archive explicit.

Secrets

manifest contains SecretRef only;
no secret value in bundle.

Side Effects

publish/deploy/webhook suppressed by default.

Replay

new RunId created;
original run remains unchanged.

Comparison

digest mismatch preserved as evidence;
normalized match distinct from bit-for-bit.

Security

revoked toolchain runs only forensic/blocked by policy.

DR

partial archive restore reduces replay availability honestly.

240. Chaos Tests

Inject:

archive object missing
CAS corruption
old image unreadable
toolchain materialization failure
provider VM timeout
compatibility bridge failure

Expected:

fidelity/gaps updated
original evidence untouched
no silent substitution

241. Scale Tests

Test:

years of run history
large monorepo source trees
large toolchain closures
many replay requests
air-gap bundle import

Historical replay creates a new Run/ReplayId.
Original Run/Job/evidence remains immutable.
Replay uses exact SourceSnapshotId when available.
Current branch state is never silently substituted.
Original PipelineIr is preferred over reparsing historical config.
Toolchains are exact/digest-bound.
Dependency blobs resolve by exact archived identity.
Missing dependencies become explicit ReplayGap.
Runner/environment substitution lowers fidelity.
Secret values are never stored in ReplayManifest.
Live external effects are suppressed by default.
Historical policy explains original decision but current policy governs execution safety.
Revoked/vulnerable artifacts can be blocked or sandboxed.
Reproduction mismatch is preserved as evidence.
Replay output cannot silently replace original artifact.
Test-data privacy restrictions can intentionally reduce replay fidelity.
Replay retention class is explicit.
CAS GC respects replay roots.
Release builds can retain full reproducibility closure.
Branch CI may have shorter replay retention.
Air-gap replay bundles verify digest/signature.
Replay bundles exclude secrets.
Federation/residency rules constrain archive movement.
DR recalculates replay availability after partial restore.
Unsupported historical metadata becomes explicit compatibility issue.
Performance replay distinguishes hardware/environment mismatch.
Forensic replay cannot access production credentials.
Tenant archive access is isolated.
Best-effort replay is labeled honestly.
Forgeyard dogfoods historical replay for its own release and incident investigations.

244. Production Readiness Gates

Do not call historical replay production-ready until:

ReplayManifest is captured automatically
source/pipeline/toolchain identities are exact
dependency archive closure is verified
secret exclusion is proven
live side effects are suppressed by default
fidelity/gap reporting is machine-enforced
release reproduction works
old metadata compatibility path exists
DR/air-gap replay tests pass
security/forensic isolation tests pass

245. Architectural Invariants

replay never mutates historical truth;
every replay has explicit fidelity;
exact source is preferred;
no current branch substitution;
archived IR preferred over reparsing;
toolchain substitutions are explicit;
dependency substitutions are explicit;
missing inputs become gaps;
secrets are references, not archived values;
live side effects are suppressed by default;
current security policy governs replay execution;
original policy remains evidence;
revoked artifacts do not regain trust through replay;
replay output is new evidence/artifact;
mismatch is preserved;
privacy may intentionally limit replay;
retention determines replay capability;
CAS GC honors replay roots;
historical compatibility is versioned;
air-gap bundles are verifiable;
forensic replay is isolated;
tenant access remains isolated;
performance replay requires environment qualification;
DR may reduce replay fidelity honestly;
archive corruption is first-class;
no fake exact replay claim;
replay works independently of current VCS branch existence;
replay does not depend on warm cache;
historical external state is never invented;
Forgeyard dogfoods its own replay system.

246. Final Target Architecture

                   Original RunId
                        │
                        ▼
                  ReplayManifestId
                        │
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
       Source        Toolchains    Dependencies
          │             │             │
          └─────────────┼─────────────┘
                        ▼
                 ReplayPlan + Fidelity
                        │
                        ▼
                    New Replay Run
                        │
                        ▼
                 Output/Evidence Compare

Fidelity:

all exact inputs retained
      ↓
deterministic replay

some equivalent substitutions
      ↓
equivalent/semantic replay

critical inputs missing
      ↓
best-effort/evidence-only/impossible

The key guarantee is:

Forgeyard can explain and reproduce historical CI behavior to the exact fidelity supported by retained evidence. It never hides missing inputs, silently upgrades historical dependencies, resurrects old secrets, or repeats destructive side effects merely to claim reproducibility.

247. Extended Architecture Sequence

01 Core Domain & Foundation
02 Storage & Metadata
03 CAS & Artifact Data Plane
04 Pipeline IR / Parsing / Planning
05 Run / Job State Machine
06 Scheduler
07 Runner / Agent
08 Sandbox & Executor
09 Transport / QUIC / Protocol
10 Events / Reconciliation
11 Policy / Authorization / Identity
12 Secrets / Trust
13 Supply Chain / SBOM / Provenance / Signing
14 Packaging
15 Release
16 Deployment
17 Observability / Health / Doctor
18 API / Axum
19 Dioxus UI
20 Device Lab
21 SCM Provider Integrations
22 HA / Coordination / Raft
23 RBE Interop
24 Plugin / Extension Architecture
25 Operations / Backup / Upgrade / DR
26 Self-Hosting / Bootstrap / Release Forgeyard
27 Multi-Tenancy / Quotas / Resource Governance
28 Audit / Compliance / Security Governance
29 Notifications / Alerting / Human Workflow
30 Entitlements / Licensing / Subscription
31 Search / Indexing / Operational Analytics
32 Test Results / Quality / Coverage / Flaky Intelligence
33 Benchmark / Performance / Load / Capacity
34 Monorepo / Dependency Graph / Affected Work
35 Developer Experience / Local Dev
36 Dependency / Registry / Mirror Governance
37 Static Analysis / Security Findings
38 Cache / Remote Cache / Correctness
39 Configuration / Feature Flags / Runtime
40 Security / Threat Model / Incident Response
41 Release Distribution / Update Delivery
42 Workflow Templates / Golden Paths
43 Runner Fleet Autoscaling / Provisioning
44 Pipeline Triggers / Schedules / Dispatch
45 Cost Accounting / FinOps
46 Data Lifecycle / Retention / Privacy
47 CI/CD Migration / Compatibility
48 Failure Diagnosis / Reproduction / Bisect
49 Service Catalog / Ownership / Developer Portal
50 Reliability / SLO / Error Budget / Resilience
51 Multi-Region Federation / Edge / Disconnected
52 Artifact Registry / OCI / Package Distribution
53 Infrastructure-as-Code / Preview Environments / Drift
54 Merge Queue / Speculative Integration / Batch Validation
55 AI-Assisted CI Optimization / Engineering Copilot
56 Test Data / Fixtures / Ephemeral Databases / Service Virtualization
57 API / ABI / Schema / Protocol Compatibility / Contract Evolution
58 Runner Image Factory / Golden Images / Patch Management / Baseline Attestation
59 Network Connectivity / Private Resource Access / Egress / Tunneling / Zero-Trust Connectivity
60 Workflow Concurrency / Distributed Locks / Idempotency / Reservations / Exclusive Coordination
61 Incident Management / On-Call / Escalation / Response Coordination / Postmortem
62 Environment Promotion / Progressive Delivery / Feature Rollout / Canary Analysis / Automated Rollback
63 Database Schema Migration / Online Backfill / Data Transformation / Zero-Downtime Change Orchestration
64 Remote Development Environments / Cloud Workspaces / Developer Workspace Orchestration
65 Build Graph Replay / Historical Reproducibility / Time-Travel CI / Evidence Reconstruction

Keyboard shortcuts

Forgeyard System Architecture & Engineering Documentation