02 — Forgeyard Storage & Metadata System Architecture
Document type: Core Infrastructure System & Architecture
Project: Forgeyard CI/CD
Subsystem: Metadata persistence, transactional state, database abstraction, standalone Stoolap storage, distributed PostgreSQL/Neon storage, schema evolution, consistency, backup/restore, tenancy, and persistence reliability
Architecture style: Store-interface driven, metadata/CAS separation, standalone-first with distributed upgrade path
Status: Target production architecture for the new Forgeyard repository
Relationship to prior work: Builds directly on 01-forgeyard-core-domain-foundation.md. It assumes the previously defined VCS-neutral source model, Change Proposal model, hermetic/reproducible build architecture, and workspace structure without redefining them.
1. Purpose
Forgeyard needs a persistence architecture that works in two very different environments:
MODE 1
Standalone / offline / local developer machine
MODE 2+
Distributed / team / enterprise / HA
The storage layer must support both without forcing:
Postgres everywhere
and without allowing:
database-specific logic everywhere
The central rule is:
Forgeyard domain services depend on typed persistence capabilities, not on Stoolap, SQL, PostgreSQL, Neon, connection pools, or database-row structures.
The second central rule is:
Metadata belongs in the metadata store. Large immutable bytes belong in CAS.
2. Architectural Position
Domain Services
│
▼
Persistence Ports
│
┌──────────┴──────────┐
▼ ▼
Stoolap Adapter PostgreSQL Adapter
│ │
▼ ▼
Local DB File PostgreSQL / Neon
Bulk objects remain separate:
metadata store
│
├── artifact metadata
├── source metadata
├── job/run state
├── policy/audit metadata
└── CAS object references
│
▼
CAS
3. Goals
The subsystem MUST:
- provide one stable persistence abstraction;
- support Stoolap for standalone mode;
- support PostgreSQL/Neon for distributed mode;
- keep database implementation details out of domain crates;
- provide explicit transactions;
- provide idempotent write patterns;
- support optimistic concurrency;
- support row/version conflict detection;
- support append-only event/audit data;
- support leases;
- support reconciliation;
- support schema migrations;
- support expand-contract upgrades;
- support backup/restore;
- support point-in-time recovery in distributed deployments where backend supports it;
- support tenancy boundaries;
- support indexes for high-volume workflows;
- separate metadata from CAS;
- provide conformance tests;
- support degraded/recovery behavior.
4. Non-Goals
The metadata store is not:
artifact blob storage
source tarball storage
build-cache object storage
container registry
large log-object store
Those belong in CAS/object storage.
The metadata store may keep:
small inline metadata
indexes
digests
references
state
audit records
5. Workspace Structure
crates/store/
├── forgeyard-store/
├── forgeyard-store-model/
├── forgeyard-store-transaction/
├── forgeyard-store-stoolap/
├── forgeyard-store-postgres/
├── forgeyard-store-migration/
├── forgeyard-store-backup/
├── forgeyard-store-health/
└── forgeyard-store-testkit/
Optional later:
├── forgeyard-store-readmodel/
├── forgeyard-store-archive/
└── forgeyard-store-metrics/
6. forgeyard-store
The primary capability crate.
It exposes:
traits
commands
queries
transaction boundaries
typed persistence errors
It MUST NOT depend on:
sqlx
tokio-postgres
Stoolap implementation internals
Neon SDK
7. forgeyard-store-model
Common persistence-neutral models.
Examples:
#![allow(unused)]
fn main() {
pub struct EntityVersion(u64);
pub struct PageRequest {
pub limit: PageSize,
pub cursor: Option<PageCursor>,
}
pub struct Page<T> {
pub items: Vec<T>,
pub next: Option<PageCursor>,
}
}8. Store Trait Strategy
Do NOT create one enormous trait with hundreds of methods if it can be avoided.
Prefer subsystem-oriented capability traits:
#![allow(unused)]
fn main() {
pub trait ProjectStore { ... }
pub trait RunStore { ... }
pub trait JobStore { ... }
pub trait RunnerStore { ... }
pub trait SourceStore { ... }
pub trait VcsMetadataStore { ... }
pub trait ChangeStore { ... }
pub trait ArtifactMetadataStore { ... }
pub trait ReleaseStore { ... }
pub trait DeploymentStore { ... }
pub trait PolicyStore { ... }
pub trait AuditStore { ... }
}Then aggregate:
#![allow(unused)]
fn main() {
pub trait ForgeyardStore:
ProjectStore
+ RunStore
+ JobStore
+ RunnerStore
+ SourceStore
+ ChangeStore
+ ArtifactMetadataStore
+ ReleaseStore
+ DeploymentStore
+ PolicyStore
+ AuditStore
+ Send
+ Sync
{}
}9. Why Capability Traits
Benefits:
smaller interfaces
easier tests
clear ownership
less accidental coupling
incremental implementation
easier future extraction
10. Transaction Model
Transactions are explicit capabilities.
#![allow(unused)]
fn main() {
#[async_trait]
pub trait TransactionManager {
type Tx: ForgeyardTransaction;
async fn begin(&self) -> Result<Self::Tx, StoreError>;
}
}11. Transaction Scope
Transactions should be used for:
state transition + event append
lease acquire + job state update
proposal revision append + stale-evidence updates
release promotion state + audit record
Do not wrap network/CAS uploads inside long SQL transactions.
12. Transaction Boundary Rule
Correct:
prepare immutable CAS object
↓
start transaction
↓
write metadata reference
↓
commit
Not:
start transaction
↓
upload 5 GB artifact
↓
commit
13. Metadata / CAS Atomicity
Because SQL and CAS are separate systems, true cross-system ACID is not assumed.
Use:
prepare
commit metadata
reconcile orphan
or:
upload CAS
verify
write metadata reference transactionally
Then GC unreferenced CAS objects after grace period.
14. Entity Versioning
Use optimistic concurrency.
#![allow(unused)]
fn main() {
pub struct Versioned<T> {
pub value: T,
pub version: EntityVersion,
}
}Updates require:
expected_version
15. Compare-and-Set Update
Conceptually:
UPDATE entity
SET ...
WHERE id = ?
AND version = expected
If no row changed:
StoreConflict
16. Why Optimistic Concurrency
It protects against:
lost updates
stale API writes
duplicate workers
concurrent reviews
queue races
provider sync races
17. Idempotency
Write operations that can be retried should accept:
#![allow(unused)]
fn main() {
pub struct IdempotencyKey(...);
}Store mapping:
scope + idempotency key -> result identity
18. Idempotent Command Example
CreateRun(
request_id = X
)
Retrying X returns same run rather than creating a duplicate.
19. Idempotency Scope
Include enough context:
tenant
operation
resource
to prevent cross-operation collision.
20. Database Error Model
#![allow(unused)]
fn main() {
pub enum StoreError {
NotFound,
Conflict,
ConstraintViolation,
TransactionAborted,
SerializationFailure,
Unavailable,
Timeout,
Corruption,
MigrationRequired,
Unsupported,
Internal,
}
}Database-specific error codes are translated inside adapters.
21. Retry Mapping
Examples:
serialization failure -> retry/backoff
deadlock -> retry/backoff
unique violation -> depends on operation
corruption -> never retry blindly
timeout -> retry/reconcile depending context
22. Standalone Mode
Standalone composition:
forgeyard
↓
forgeyard-store-stoolap
↓
local DB file
No Postgres requirement.
23. Why Stoolap in Mode 1
Desired properties:
embedded
zero external daemon
local filesystem
simple install
transaction support
Rust integration
low operational burden
24. Stoolap Adapter
crates/store/forgeyard-store-stoolap/
├── Cargo.toml
└── src/
├── lib.rs
├── db.rs
├── schema.rs
├── transaction.rs
├── migration.rs
├── project.rs
├── run.rs
├── job.rs
├── runner.rs
├── source.rs
├── vcs.rs
├── change.rs
├── artifact.rs
├── release.rs
├── deployment.rs
├── policy.rs
├── audit.rs
├── health.rs
└── error.rs
25. Stoolap Database Location
Default local path should be under Forgeyard data directory:
~/.local/share/forgeyard/
or platform-equivalent application data directory.
Example:
forgeyard.db
Do not place production state in current working directory by default.
26. Standalone Locking
Only one standalone daemon/process should own mutable local DB state unless Stoolap safely supports required multi-process access.
Use:
process lock
or
DB-supported exclusive coordination
to avoid split-brain local writers.
27. Local Backup
Standalone backup command:
forgeyard storage backup
should produce:
metadata snapshot
CAS manifest
configuration manifest
version info
Optional:
include CAS bytes
for full portable backup.
28. Distributed Mode
Distributed composition:
forgeyard-daemon
↓
forgeyard-store-postgres
↓
PostgreSQL / Neon
29. PostgreSQL Authority
In distributed Forgeyard:
Postgres/Neon
is authoritative for shared metadata.
Raft does not replace it.
30. PostgreSQL Adapter Tree
crates/store/forgeyard-store-postgres/
├── Cargo.toml
└── src/
├── lib.rs
├── pool.rs
├── connection.rs
├── transaction.rs
├── statement.rs
├── retry.rs
├── row/
│ ├── project.rs
│ ├── run.rs
│ ├── job.rs
│ ├── runner.rs
│ ├── source.rs
│ ├── vcs.rs
│ ├── change.rs
│ ├── artifact.rs
│ ├── release.rs
│ ├── deployment.rs
│ ├── policy.rs
│ └── audit.rs
├── query/
│ ├── project.rs
│ ├── run.rs
│ ├── job.rs
│ ├── runner.rs
│ ├── source.rs
│ ├── vcs.rs
│ ├── change.rs
│ ├── artifact.rs
│ ├── release.rs
│ ├── deployment.rs
│ ├── policy.rs
│ └── audit.rs
├── health.rs
└── error.rs
31. Row Types
DB row structs remain adapter-local.
Bad:
#![allow(unused)]
fn main() {
pub struct Run {
pub created_at: sqlx::types::chrono::DateTime<Utc>,
}
}inside domain.
Correct:
PostgresRunRow
↓ TryFrom
Run
32. SQL Placement
All SQL stays inside:
forgeyard-store-postgres
or migration files.
No SQL strings in:
scheduler
change service
release service
VCS service
33. Query Ownership
Each domain store capability owns its persistence queries.
Example:
RunStore
↔
query/run.rs
34. Connection Pool
Pool configuration:
min connections
max connections
acquire timeout
idle timeout
max lifetime
Must be observable.
35. Pool Size
Do not set huge default pools.
Distributed capacity should consider:
daemon replicas × pool max
against database connection limits.
36. Neon Considerations
Neon-compatible behavior should treat PostgreSQL protocol/semantics as source of truth.
Forgeyard should avoid depending on Neon-only semantics in the domain layer.
Neon integration can optimize:
connection setup
branch/testing workflows
serverless pooling considerations
through deployment/configuration rather than core domain coupling.
37. Connection Resilience
Transient connection loss:
retry with bounded backoff
but state-changing operations must remain idempotent or transactional.
38. Transaction Isolation
Default recommendations depend on operation.
General CRUD:
READ COMMITTED
Critical compare/update workflows:
explicit row version/CAS
Certain queue/lease operations may require:
SELECT ... FOR UPDATE
or equivalent safe transactional primitives.
39. Serializable Transactions
Use only where necessary.
Do not make the entire system SERIALIZABLE by default.
For workflows requiring serializable behavior:
retry serialization failures
explicitly.
40. Lease Storage
Lease record:
#![allow(unused)]
fn main() {
pub struct LeaseRecord {
pub lease_id: LeaseId,
pub owner: RunnerId,
pub resource: LeaseResource,
pub issued_at: Timestamp,
pub expires_at: Timestamp,
pub epoch: LeaseEpoch,
}
}41. Lease Acquisition
Transactionally:
verify resource eligible
create lease
move resource state
append event
commit
42. Lease Expiry
DB expiry is not magic.
Use reconcilers:
find expired leases
↓
verify current state
↓
mark lost/requeue
43. Job Persistence
Core job row metadata:
job_id
run_id
state
attempt
priority
created_at
updated_at
version
Large logs are not stored inline.
44. Run Persistence
Run metadata:
run_id
project_id
pipeline_id
source_snapshot
status
created_by
timestamps
version
45. Source Metadata Persistence
Store:
repository ID
native revision ID
SourceSnapshotId
provenance ID
nested-source relationships
signature evidence references
Source bytes remain CAS.
46. Change Proposal Persistence
Store:
proposal
proposal revisions
reviews
approvals
comments
checks
policy decisions
integration candidates
queue state
provider binding
Bulk diff/rendered artifacts may live in CAS.
47. Artifact Metadata Persistence
Store:
artifact ID
digest
CAS object ID
type
size
producer
retention
created_at
Not artifact bytes.
48. Release Persistence
Store:
release ID
candidate artifact digests
approval state
signing evidence
promotion state
timestamps
49. Deployment Persistence
Store:
deployment ID
release/artifact
environment
state
health evidence
rollback pointer
50. Audit Persistence
Audit is append-only.
audit_event_id
actor
action
resource
timestamp
payload digest/reference
Do not allow ordinary update/delete semantics.
51. Event Persistence
Durable domain events may use:
event_log table
or subsystem-specific append tables.
Avoid building a giant generalized event-sourcing framework unless needed.
52. Event Outbox
For reliable external/internal publication:
domain transaction
↓
state update
+ outbox row
↓
commit
↓
publisher sends event
↓
mark delivered
53. Why Outbox
Prevents:
DB commit succeeded
but event publish failed
from losing important state notifications.
54. Outbox Semantics
At-least-once.
Consumers must deduplicate by:
EventId
55. Inbox / Dedup
For inbound durable commands/events:
processed_message
table can record:
message ID
consumer
processed_at
result
56. Multi-Tenancy
Every shared metadata entity should be tenant-scoped where applicable.
Examples:
tenant_id
organization_id
project_id
57. Tenant Boundary Rule
Queries must not rely solely on caller filtering.
Storage API should include tenant context.
Bad:
#![allow(unused)]
fn main() {
get_project(project_id)
}Better:
#![allow(unused)]
fn main() {
get_project(tenant_id, project_id)
}or a scoped store/session.
58. Scoped Store
Possible abstraction:
#![allow(unused)]
fn main() {
pub struct TenantStore<'a> {
tenant: TenantId,
inner: &'a dyn ForgeyardStore,
}
}59. Row-Level Security
Postgres RLS may be considered later.
But domain correctness must not rely exclusively on RLS.
Use both:
typed tenant scoping
+
DB constraints
60. Foreign Keys
Use foreign keys where they preserve real invariants.
Examples:
job -> run
run -> project
proposal revision -> proposal
artifact -> project
61. Cascades
Avoid broad destructive ON DELETE CASCADE for audit/history-critical data.
Deletion should usually be controlled by service policy.
62. Soft Delete
Use only where domain semantics require retention.
Avoid automatic deleted_at on everything.
63. Immutable Records
Examples:
audit events
proposal revisions
source provenance
release attestations
should be append-only/immutable.
64. Mutable Records
Examples:
runner heartbeat
queue position
current job state
may update in place with versioning.
65. Append + Current Projection Pattern
For important state:
append history/event
+
update current row
within same transaction.
66. Index Strategy
Indexes should support real query patterns.
Examples:
jobs by state/priority
jobs by run
runs by project/created_at
runners by availability
proposals by repository/status
queue entries by target/state
audit by tenant/time
outbox by delivery state
67. Avoid Over-Indexing
Every index costs:
write amplification
storage
vacuum/maintenance
Add based on measured query needs.
68. Pagination
Use cursor/keyset pagination for large tables.
Avoid deep:
OFFSET 100000
queries.
69. Cursor Model
#![allow(unused)]
fn main() {
pub struct PageCursor(BoundedString);
}Opaque to clients.
70. Ordering
Pagination order must be stable.
Example:
created_at DESC, id DESC
71. Search
Do not turn primary metadata DB into universal full-text/search engine initially.
Start with indexed structured filters.
Add dedicated search capability only when needed.
72. Logs
High-volume logs should not live as giant SQL rows.
Store:
log stream metadata
sequence ranges
CAS/object references
73. Log Metadata
Possible:
job_id
stream_id
first_seq
last_seq
object_ref
created_at
74. Log Tail Index
Small recent/tail buffers may be held in memory or optimized store, but durable full logs remain separate.
75. Metrics
Metrics are observability data, not core transactional metadata.
Do not store time-series metrics in primary Forgeyard metadata DB by default.
76. Health
Storage health check:
connection
read
write optional
migration status
latency
pool saturation
replica lag where relevant
77. Readiness
Daemon should not report ready if:
required migrations pending
DB inaccessible
required schema incompatible
78. Schema Migration Architecture
Workspace:
crates/store/forgeyard-store-migration/
migrations/
79. Migration Version
Every migration has stable ID:
0001
0002
...
Never rewrite applied migrations.
80. Migration Metadata
Track:
migration ID
checksum
applied_at
Forgeyard version
81. Migration Checksum
If an applied migration file changes unexpectedly:
fail
Do not silently accept history rewrite.
82. Expand-Contract
Distributed rolling upgrade strategy:
1. expand schema
2. deploy code using both/new fields
3. backfill
4. verify
5. switch reads
6. contract old schema later
83. Breaking Migration Rule
Never require all daemons/agents to stop simultaneously unless explicitly performing maintenance.
84. Migration Lock
Only one migration runner should apply schema changes.
Use:
database advisory lock
or
coordination lock
85. Automatic Migration
Production default:
safe expand-only migrations may auto-run
Potential destructive/expensive migrations:
explicit operator command
86. Backfill
Large backfills should be:
chunked
resumable
observable
idempotent
Not one huge transaction.
87. Backfill State
Store progress:
migration ID
cursor
processed rows
errors
updated_at
88. Schema Ownership
Each subsystem owns its tables logically.
Example:
change_*
vcs_*
run_*
artifact_*
Avoid one global miscellaneous table design.
89. Database Namespace
Optionally use Postgres schemas:
forgeyard_core
forgeyard_change
forgeyard_vcs
but this can increase migration complexity.
Initial recommendation:
single DB schema + disciplined prefixes
unless operational need justifies multiple schemas.
90. Naming Convention
Tables:
snake_case plural
Columns:
snake_case
Primary keys:
<entity>_id
Version:
entity_version
91. ID Storage
ULID/entity IDs can be stored as:
UUID-compatible binary
byte representation
text
Choose one canonical DB representation.
Recommendation:
native UUID where compatible with chosen ID semantics
or fixed binary
Avoid arbitrary variable strings if not needed.
92. Digest Storage
Store:
algorithm
digest bytes
or canonical fixed binary columns.
Do not repeatedly store long textual prefixes when unnecessary.
93. Timestamps
Use database timestamp types carefully.
Domain converts to/from:
forgeyard-time::Timestamp
94. Server-Side Now
Avoid relying on DB NOW() as sole domain time source when cross-system consistency matters.
Use explicit timestamps from service Clock, unless DB timestamp is intentionally authoritative for specific internal purpose.
95. Transaction Timestamp
A transaction can use one consistent service timestamp:
occurred_at
for state + event.
96. Sequence Numbers
Use explicit sequence values for:
logs
event streams
proposal revision number
job attempts
where ordering matters.
97. Job Attempts
Separate:
job
job_attempt
A job may have multiple attempts.
98. Attempt Table
Store:
attempt_id
job_id
attempt_number
runner_id
lease_id
state
start/end
failure classification
99. Runner Heartbeats
High-frequency writes can overload DB.
Architecture options:
coalesced heartbeat writes
in-memory recent state + periodic durable update
separate lightweight heartbeat table
Do not write every tiny heartbeat update into a huge row with many indexed columns.
100. Runner Liveness
Separate:
reported_at
lease heartbeat
health status
from immutable runner identity/capabilities.
101. Capability Storage
Runner capability blob should be:
versioned
canonical
digest-addressed if large
with queryable important fields extracted/indexed.
102. Scheduler Query Path
Avoid scheduler scanning entire runner/job tables.
Maintain efficient queries/indexes for:
eligible jobs
available runners
leases
resource availability
103. Queue Query Path
Integration queue tables need:
target
state
priority
sequence
created_at
indexes.
104. Advisory Locks
Postgres advisory locks can be used for:
migration lock
rare singleton operation
but do not make them the universal distributed lock architecture.
Forgeyard coordination layer handles broader leadership/exclusive semantics.
105. Postgres SKIP LOCKED
May be useful for certain work-claim patterns.
But scheduler semantics should remain explicit and tested.
Do not let a SQL queue become the scheduler architecture.
106. Database as Coordination
Use DB transaction semantics where natural.
Do not use DB as replacement for:
agent protocol
runner leases
Raft leadership
CAS transfer
107. Neon Branching
Neon branching may be useful for:
test environments
migration tests
ephemeral integration validation
But it remains an optional deployment optimization, not core requirement.
108. Read Replicas
Read replicas may serve:
analytics/UI historical reads
audit browsing
Never serve consistency-sensitive write-after-read decisions without explicit consistency model.
109. Replica Lag
If replica used:
lag must be observable
and critical paths must use primary/consistent reads.
110. Read Consistency Classes
Possible API:
#![allow(unused)]
fn main() {
pub enum ReadConsistency {
Strong,
Eventual,
}
}Use sparingly.
Most domain stores can choose internally based on operation.
111. Strong Reads
Required for:
lease validation
mergeability submit
approval current revision
release promotion
permission changes
112. Eventual Reads
May be acceptable for:
dashboard counters
historical lists
analytics
113. Backup Architecture
Distributed backup includes:
database backup/PITR
CAS backup/replication
configuration backup
secret-provider backup according to provider
114. Metadata Backup
Postgres:
managed backup
WAL/PITR
logical dump for portability
depending deployment.
Neon can provide managed PostgreSQL recovery capabilities, but Forgeyard should still define its own restore/runbook expectations.
115. Stoolap Backup
Provide application-consistent backup operation.
Do not recommend copying live DB file unless backend guarantees safety.
116. CAS Backup
CAS backup is separate.
Metadata backup alone may leave artifact references without bytes.
117. Backup Manifest
Forgeyard backup manifest:
#![allow(unused)]
fn main() {
pub struct BackupManifest {
pub forgeyard_version: String,
pub schema_version: StoreSchemaVersion,
pub created_at: Timestamp,
pub metadata_backup: BackupObjectRef,
pub cas_manifest: Option<CasBackupManifestRef>,
}
}118. Restore
Restore stages:
validate backup
restore metadata
validate schema
restore/attach CAS
run integrity verification
reconcile
resume service
119. Restore Into Newer Version
Preferred:
restore into matching schema/version
then migrate forward
not arbitrary direct restore into incompatible latest schema.
120. Restore Verification
Verify:
foreign keys
critical counts
CAS references
release artifacts
source snapshots
audit chain
121. Disaster Recovery
After outage:
DB restored
↓
CAS checked
↓
leases expired/reconciled
↓
running jobs marked lost/retried
↓
provider state reconciled
↓
queues revalidated
122. Lease Recovery
Never assume a restored “Running” job is still running.
Reconcile via:
agent heartbeat
lease expiry
attempt identity
123. Change Queue Recovery
Restored integration candidate must revalidate:
target revision still expected
before submission.
124. Release Recovery
Release promotion state must verify:
signed artifact exists
provenance exists
target registry/store state
before retry.
125. Data Integrity Checker
Tool/service:
forgeyard storage verify
checks:
dangling foreign keys
invalid states
CAS refs missing
duplicate immutable mappings
revision->snapshot mutation
release artifact mismatch
126. Database Constraints
Use DB constraints for invariants that are easy and stable:
NOT NULL
UNIQUE
FOREIGN KEY
CHECK
Domain validates too.
Defense in depth.
127. Unique Constraints
Examples:
repository + native revision -> snapshot mapping
proposal + revision_number
job + attempt_number
tenant + idempotency scope/key
128. Immutable Mapping Constraint
Critical:
same VCS revision cannot map to different SourceSnapshotId
unless backend semantics explicitly say identity is mutable, which should be treated separately.
129. State Constraints
DB may use CHECK constraints for simple legal values if representation allows.
But transition legality belongs in domain.
130. Store Conformance Testkit
crates/store/forgeyard-store-testkit/
Every backend runs same test suite.
131. Conformance Areas
CRUD
transactions
optimistic concurrency
idempotency
pagination
tenant isolation
leases
append-only records
migration compatibility
error mapping
132. Example Conformance Test
#![allow(unused)]
fn main() {
async fn optimistic_update_rejects_stale_version<S: RunStore>(store: &S) {
...
}
}Run against:
Stoolap
Postgres
133. Ephemeral Postgres Tests
Integration tests may use:
temporary local Postgres
containerized Postgres
test database
Forgeyard should not require Neon for correctness testing.
134. Neon Compatibility Tests
Separate optional test suite validates:
connection behavior
pooling
migrations
long-running transactions assumptions
against Neon.
135. Stoolap Compatibility Tests
Run on every PR because standalone mode is first-class.
136. Migration Tests
For every migration:
old schema fixture
↓
apply migration
↓
read/write using new code
137. N/N-1 Tests
Distributed rolling upgrade tests:
N-1 daemon schema compatibility
N daemon
shared DB
where supported.
138. Performance Benchmarks
Measure:
job enqueue
lease acquisition
heartbeat write
proposal read
queue read
audit append
pagination
139. Load Tests
Simulate:
100k jobs
10k runners
large audit history
large proposal histories
many concurrent queue writes
Targets can evolve, but architecture must be tested under realistic scale.
140. Query Instrumentation
Every DB query should record:
latency
operation name
rows
errors
timeout
Do not log SQL with secrets.
141. Slow Query Detection
Metrics:
p50/p95/p99
for named operations.
142. Query Naming
Every query path gets stable logical name:
run.get
job.claim
change.current
queue.next
audit.append
143. Connection Metrics
Expose:
pool size
in-use
idle
waiters
acquire latency
errors
144. Storage Health Status
#![allow(unused)]
fn main() {
pub enum StorageHealth {
Healthy,
Degraded,
Unavailable,
MigrationRequired,
}
}145. Degraded Mode
Some reads may continue if DB is degraded.
But do not allow unsafe state-changing operations without authoritative metadata storage.
146. Read-Only Emergency Mode
Optional operator mode:
metadata reads
artifact downloads
audit browsing
while writes disabled.
147. Database Credentials
Use SecretRef.
Do not store:
password in config file
148. TLS
Postgres production connections should support:
TLS verification
custom CA
client certificate where applicable
149. Credential Rotation
Pool must reconnect safely after secret rotation.
Avoid requiring daemon restart where possible.
150. Database URL Logging
Always redact credentials.
151. SQL Injection
Use parameterized queries only.
Dynamic identifiers require strict whitelist/typed construction.
152. Migration Security
Migration files are trusted code.
Require:
review
digest verification
release provenance
153. Tenant Query Safety
Architecture checker/testing should discourage adapter methods lacking tenant scoping for tenant-owned resources.
154. Audit for Admin Storage Operations
Record:
migration
backup
restore
repair
manual data correction
155. Manual Data Repair
Provide controlled command:
forgeyard storage repair
Never recommend direct ad-hoc SQL as normal operational workflow.
156. Repair Plan
detect
generate plan
operator approve
apply transactionally
audit
verify
157. Data Export
Provide structured export for:
project metadata
audit
release provenance
without exporting secret values.
158. Data Portability
Store abstraction should make future migration possible.
But do not target lowest-common-denominator DB semantics so aggressively that Postgres capabilities cannot be used inside adapter safely.
159. Backend-Specific Optimizations
Allowed inside adapter:
Postgres indexes
JSONB for non-authoritative extension metadata
generated columns
partial indexes
advisory locks
if core semantics remain stable.
160. JSONB Rule
Use JSONB only for:
provider payload fragments
extension metadata
rare unstructured evidence
Do not put core typed domain state into opaque JSON blobs.
161. Binary Metadata
Postcard blobs may be stored for:
versioned opaque evidence
but critical searchable state should use structured columns.
162. RON in Database
Do not use RON as primary DB row format.
RON is for human configuration.
163. CAS Reference Type
#![allow(unused)]
fn main() {
pub struct CasObjectRef {
pub digest: Digest,
pub media_type: Option<MediaType>,
pub size: ByteSize,
}
}164. Artifact Record Example
#![allow(unused)]
fn main() {
pub struct ArtifactMetadata {
pub id: ArtifactId,
pub project: ProjectId,
pub object: CasObjectRef,
pub producer: JobId,
pub created_at: Timestamp,
pub retention: RetentionPolicy,
}
}165. Source Snapshot Record Example
#![allow(unused)]
fn main() {
pub struct SourceSnapshotRecord {
pub id: SourceSnapshotId,
pub root_tree: TreeObjectId,
pub provenance: SourceProvenanceId,
pub created_at: Timestamp,
}
}166. Change Proposal Record Example
#![allow(unused)]
fn main() {
pub struct ChangeProposalRecord {
pub id: ChangeProposalId,
pub repository: RepositoryId,
pub lifecycle: ProposalLifecycle,
pub current_revision: ProposalRevisionId,
pub version: EntityVersion,
}
}167. Store Command vs Row Model
Public store traits should accept domain-oriented command/record types.
Adapter-specific rows remain private.
168. Transactional State + Audit
Pattern:
validate transition
↓
BEGIN
↓
update state with expected version
↓
append audit/event
↓
COMMIT
169. Transactional State + Outbox
Pattern:
BEGIN
↓
state
audit
outbox
↓
COMMIT
Then async delivery.
170. Exactly-Once Warning
Do not claim exactly-once distributed effects.
Database can provide exactly-once row mutation in one transaction.
Cross-system delivery remains:
at-least-once + idempotency + reconciliation
171. Store Service Boundary
A service should depend on:
#![allow(unused)]
fn main() {
Arc<dyn RunStore>
}or aggregated application storage interface.
Not:
#![allow(unused)]
fn main() {
PgPool
}172. Composition Wiring
Standalone:
StoolapStore
↓
Arc<dyn ForgeyardStore>
Distributed:
PostgresStore
↓
Arc<dyn ForgeyardStore>
Domain services unchanged.
173. Avoid Generic Repository Pattern Abuse
Do not create:
#![allow(unused)]
fn main() {
Repository<T>
}for everything.
Persistence interfaces should express domain operations.
Example:
#![allow(unused)]
fn main() {
acquire_job_lease(...)
}can be better than generic:
#![allow(unused)]
fn main() {
update(job)
}because it encodes concurrency semantics.
174. Domain-Specific Store Operations
Good:
append_proposal_revision
acquire_job_lease
record_job_attempt
mark_check_stale
claim_outbox_batch
These make atomicity explicit.
175. Batch Operations
Store APIs should support efficient batch writes for:
events
artifact refs
source tree metadata
test results
where appropriate.
176. Large IN Queries
Use batch/temp-table strategies if large sets become common.
Do not generate thousands of scalar queries.
177. N+1 Query Prevention
Service tests/metrics should detect common N+1 patterns.
178. Prepared Statements
Use driver-supported prepared statement/cache behavior safely.
179. Statement Timeout
Critical DB calls should have bounded timeout.
Different operation classes may have different limits.
180. Long-Running Reports
Run historical/report queries separately from critical scheduler transaction paths.
181. Operational Separation
At scale consider:
primary operational Postgres
read replica/analytics export
not immediately another database.
182. Audit Export
SIEM export should consume event/audit stream asynchronously.
Do not make audit transaction wait for SIEM network.
183. GC Metadata
CAS GC roots live in metadata:
artifact refs
release refs
source refs
pinned refs
retention refs
184. GC Race Protection
GC must not delete object being attached concurrently.
Use:
grace periods
mark epochs
metadata snapshot
185. Retention Policy
Metadata includes:
retain until
pin
legal hold
release permanent
186. Legal Hold
Legal/compliance hold overrides normal deletion/GC.
187. Project Deletion
Deletion workflow:
mark deletion requested
verify holds
disable new writes
delete/expire owned metadata
schedule CAS GC
audit
Avoid instant destructive cascade.
188. User Deletion
Identity/privacy-related deletion may require anonymization while preserving audit integrity.
Handled by identity/compliance layer with store support.
189. Store Schema Documentation
docs/architecture/storage.md should document logical tables/entities.
SQL migration files remain implementation truth for exact DB schema.
190. ER Diagram
Generate from schema/tooling where useful.
Do not manually maintain giant diagrams that drift.
191. Schema Generator
Optional:
tools/forgeyard-schema-gen/
can produce:
ER diagram
table docs
migration status
192. Storage CLI
forgeyard storage status
forgeyard storage doctor
forgeyard storage migrate
forgeyard storage verify
forgeyard storage backup
forgeyard storage restore
forgeyard storage repair
forgeyard storage stats
193. storage status
Shows:
backend
schema version
migration state
health
pool
database size
critical table counts
194. storage doctor
Checks:
connectivity
permissions
schema
migration lock
latency
required extensions
timezone assumptions
CAS reference integrity sample
195. storage migrate
Supports:
plan
apply
status
Dangerous operations require explicit operator confirmation in CLI.
196. storage verify
Modes:
quick
full
Quick:
schema
constraints
sample CAS refs
Full:
all CAS refs
immutable mappings
audit integrity
197. storage stats
Displays:
table sizes
row counts
growth
outbox backlog
old leases
orphan references
198. Admin API
Do not expose destructive storage admin operations to ordinary public REST API by default.
Prefer:
local CLI
restricted admin API
199. Monitoring Alerts
Alert on:
DB unavailable
pool exhaustion
migration mismatch
outbox backlog
replica lag
backup stale
PITR disabled where required
corruption signal
200. Backup SLO
Enterprise deployments should define:
RPO
RTO
instead of saying merely “backups enabled.”
201. Example RPO/RTO
Example target:
RPO: 5 minutes
RTO: 30 minutes
Actual values are deployment policy, not hardcoded architecture.
202. Mode 1 Recovery
Standalone priorities:
simple backup
portable restore
minimal operator knowledge
203. Mode 2 Recovery
Distributed priorities:
PITR
tested restore
CAS consistency
reconciliation
204. Restore Drill
Production readiness requires periodic restore test.
Backup without tested restore is insufficient.
205. Migration Rollback
Schema rollback is not always safe.
Prefer forward-fix.
For app binary rollback:
expand-contract schema
keeps N-1 compatibility.
206. Destructive Schema Changes
Only after:
old code removed
backfill verified
retention window elapsed
207. Database Version Compatibility
Daemon startup checks:
min supported schema
max supported schema
Refuse unsafe mismatch.
208. Agent Independence
Agents should not connect directly to Forgeyard metadata DB.
They communicate with daemon over protocol.
This preserves trust/control boundaries.
209. CLI DB Access
Normal CLI should use API.
Direct DB admin access only through explicit maintenance commands/tools.
210. UI DB Access
Never.
Dioxus UI only calls Forgeyard APIs.
211. Provider Adapter DB Access
Provider adapters use store/service interfaces.
No direct table access outside store adapter.
212. VCS Adapter DB Access
Same rule.
VCS adapters can emit metadata/provenance to VCS service, which uses store interfaces.
213. Ecosystem Adapter DB Access
None.
Ecosystem adapters produce plans/evidence.
Orchestrating services persist through store.
214. Scheduler DB Access
Scheduler may use domain-specific scheduler/store capability interfaces.
Do not hand it raw pool.
215. Scheduler Hot Path
Hot path should minimize round trips.
Example:
get eligible batch
get runner candidates
transactionally lease
216. Queue Hot Path
Integration queue should support:
claim next entries
lease queue operation
verify target state
with minimal DB contention.
217. Lock Ordering
Define consistent row/resource lock ordering for workflows touching multiple entities.
This reduces deadlocks.
218. Deadlock Handling
If DB deadlock occurs:
translate
retry bounded
do not panic.
219. Transaction Retry Helper
Crate:
forgeyard-store-transaction
can provide:
#![allow(unused)]
fn main() {
retry_transaction(policy, || async { ... })
}only for operations proven idempotent at transaction boundary.
220. Retry Safety
Never auto-retry transaction if it contains irreversible external side effect.
221. Side-Effect Staging
Correct:
DB reserves operation
commit
external effect
record outcome
reconcile
for:
publish
sign
deploy
222. Saga-Like Workflows
Long workflows:
release
deployment
integration
use persisted state machines + compensating/reconciliation actions, not giant DB transaction.
223. Storage Data Classes
Classify:
transactional metadata
immutable provenance
audit
ephemeral coordination
high-volume stream metadata
This guides schema/index/retention.
224. Ephemeral Coordination Data
Examples:
leases
heartbeats
temporary claims
may have aggressive cleanup.
225. Historical Metadata
Examples:
releases
proposals
audit
source provenance
long retention.
226. Cleanup Jobs
Periodic:
expired idempotency entries
old ephemeral leases
delivered outbox
temporary rows
with retention windows.
227. Outbox Retention
Delivered outbox may be compacted after audit-safe period.
Do not let table grow forever.
228. Vacuum / Maintenance
Postgres operational docs should monitor:
autovacuum
bloat
long transactions
dead tuples
Forgeyard app should avoid creating pathologically long transactions.
229. Long Transaction Alert
Trace any transaction above threshold.
230. Isolation in Tests
Each integration test should use:
fresh database/schema/transaction
to avoid cross-test state.
231. Deterministic Fixtures
Store testkit uses deterministic IDs/time.
232. Fault Injection
Test adapter behavior under:
connection loss
timeout
serialization failure
deadlock
constraint violation
disk full simulation where possible
233. Corruption Simulation
For local DB, test:
invalid file
partial backup
unsupported schema
and ensure errors are clear.
234. Store API Documentation
Every method documents:
atomicity
consistency
idempotency
expected conflicts
retry behavior
235. Example Store API
#![allow(unused)]
fn main() {
#[async_trait]
pub trait JobStore {
async fn insert_job(
&self,
job: NewJobRecord,
) -> Result<Versioned<JobRecord>, StoreError>;
async fn acquire_lease(
&self,
request: AcquireJobLease,
) -> Result<JobLeaseResult, StoreError>;
async fn complete_attempt(
&self,
request: CompleteJobAttempt,
) -> Result<JobRecord, StoreError>;
}
}236. Why Domain Operations in Store
acquire_lease() needs one atomic DB operation.
A generic CRUD abstraction would encourage unsafe multi-step service logic.
237. Store Interface Granularity
Do not put business policy in store.
Store handles:
atomic persistence semantics
constraints
query efficiency
Service handles:
business decision
policy
authorization
238. Authorization Boundary
Store does not decide whether user is allowed to act.
Authz service decides.
Store enforces tenant/resource constraints.
239. Audit Boundary
Security-sensitive service writes audit event in same transaction where possible.
240. Immutable Audit Payload
If audit payload is large:
CAS object
+
digest/reference in DB
241. CAS Missing Object
Metadata can temporarily reference missing CAS only in controlled transitional state.
Normal committed artifact should require CAS verification before metadata becomes visible.
242. Artifact Visibility States
#![allow(unused)]
fn main() {
pub enum ArtifactVisibilityState {
Pending,
Available,
Quarantined,
Deleted,
}
}243. Upload Workflow
create upload intent
↓
upload CAS
↓
verify digest
↓
transaction: artifact metadata = Available
244. Orphan CAS Cleanup
Failed upload metadata leaves CAS object unreferenced.
GC handles after grace.
245. Missing CAS Reconciliation
If metadata says Available but object absent:
mark degraded/quarantined
alert
attempt replica recovery
246. Source Snapshot Visibility
A source snapshot should be registered only after its tree/blob closure is complete/verified.
247. Immutable Provenance Mapping
Store should reject:
same SourceSnapshotId with different canonical tree
by construction/digest verification.
248. Check Evidence Storage
Check metadata in DB:
state
kind
required
run ID
evidence refs
Large report in CAS.
249. Policy Decision Storage
Store:
policy digest
decision
violations
exception refs
evaluation timestamp
250. Data Encryption
At rest encryption generally delegated to:
filesystem/disk
managed Postgres
object store
Forgeyard must not invent custom transparent DB crypto casually.
Secrets use dedicated secret system.
251. Sensitive Metadata
Some metadata may still be sensitive:
private repo URLs
audit data
user identifiers
Protect through:
DB access control
TLS
RBAC
backup encryption
252. Data Classification
Optional model:
#![allow(unused)]
fn main() {
pub enum DataClassification {
Public,
Internal,
Confidential,
Restricted,
}
}Can drive retention/export policy later.
253. Export Policy
Admin exports respect classification.
254. SQL Logging
Never log bind values by default in production.
255. PII Minimization
Store only user/profile fields needed for operation.
Identity provider remains source for broader profile data when possible.
256. Audit Immutability
Administrative repair of audit data should be extremely restricted and separately recorded.
257. Schema Tests
Automated tests verify:
foreign keys present
unique constraints present
expected indexes present
migration checksums
258. Store Architecture Checker
Potential static check ensures:
no sqlx dependency outside postgres adapter
no Stoolap dependency outside Stoolap adapter
259. Cargo Dependency Rule
sqlx / Postgres crate allowed only in:
forgeyard-store-postgres
migration/admin helpers
not domain.
260. Stoolap Dependency Rule
Stoolap crate allowed only in:
forgeyard-store-stoolap
and its test helpers.
261. DB Serialization Types
Adapters convert:
DB integer -> typed enum
DB bytes -> typed ID/digest
DB timestamp -> forgeyard Timestamp
Conversion is fallible.
262. Invalid DB Value
If DB contains invalid enum/state:
StoreError::Corruption
not default fallback.
263. Store Startup
Startup order:
load config
resolve secret
connect
check DB version
check migrations
run allowed migrations
verify health
start services
264. Standalone Startup
ensure data dir
acquire process lock
open Stoolap
migrate
verify
start
265. Distributed Startup
connect pool
acquire migration lock if needed
schema check
migrate
health/readiness
266. Migration Ownership in HA
One daemon migrates.
Others wait/read migration state.
267. Zero-Downtime Upgrade Example
v1 schema
↓ expand migration
v1 + v2 code compatible
↓ deploy v2
backfill
↓
later contract migration
268. Store Version Compatibility
#![allow(unused)]
fn main() {
pub struct StoreCompatibility {
pub min_schema: StoreSchemaVersion,
pub max_schema: StoreSchemaVersion,
}
}269. Store Health API
#![allow(unused)]
fn main() {
pub trait StoreHealth {
async fn health(&self) -> StorageHealthReport;
}
}270. Health Report
#![allow(unused)]
fn main() {
pub struct StorageHealthReport {
pub status: StorageHealth,
pub schema: StoreSchemaVersion,
pub latency: Duration,
pub details: Vec<HealthDetail>,
}
}271. Testkit Structure
forgeyard-store-testkit/src/
├── lib.rs
├── project.rs
├── run.rs
├── job.rs
├── runner.rs
├── source.rs
├── change.rs
├── artifact.rs
├── release.rs
├── transaction.rs
├── idempotency.rs
├── tenant.rs
└── migration.rs
272. Benchmark Structure
benches/
├── job_claim.rs
├── run_create.rs
├── proposal_read.rs
├── audit_append.rs
└── outbox_claim.rs
273. Migration Directory
migrations/postgres/
├── 0001_core.sql
├── 0002_runs_jobs.sql
├── 0003_sources_vcs.sql
├── 0004_changes.sql
├── 0005_artifacts.sql
├── 0006_release_deploy.sql
├── 0007_audit_outbox.sql
└── ...
Stoolap migration representation should map equivalent domain schema semantics.
274. Logical Schema Groups
core
project
run/job
runner
source/vcs
change proposal
artifact/cache refs
release/deployment
policy
audit/events
coordination metadata
275. Store Module Ownership
Each subsystem defines:
domain models
store trait
but exact implementation sits under store adapters.
Alternative layout may keep trait near domain crate.
Recommended rule:
domain-facing persistence interface near domain
adapter implementation under store backend
Avoid circular dependencies.
276. Preferred Interface Placement
Example:
forgeyard-run-store-api
or:
forgeyard-run/src/store.rs
depending complexity.
The workspace structure can evolve, but adapter independence must remain.
277. No ORM Domain Leakage
ORM/driver annotations stay adapter-local.
278. Query Builder Choice
Use a mature Rust Postgres client/query approach.
But architecture is independent of exact library.
Do not make query library part of domain model.
279. SQL Compile-Time Checking
If using a library with compile-time query checking, integrate it into CI carefully.
Do not require production DB connectivity just to compile release unless offline query metadata is supported.
280. Prepared Query Metadata
If used, keep generated metadata versioned/updated by tooling.
281. Store Doctor Categories
REQUIRED
OPTIONAL
PERFORMANCE
SECURITY
Example:
REQUIRED: schema compatible
PERFORMANCE: missing recommended index
SECURITY: TLS disabled in production
282. Local Dev DB Reset
Command:
forgeyard dev storage reset
only for development profiles.
Never expose destructive reset in production without strong safety.
283. Fixture Loading
Test/development only:
forgeyard dev storage seed
284. Schema Documentation Generation
Generate:
docs/reference/database-schema.md
from migrations/schema tooling where possible.
285. Capacity Planning
Important metadata dimensions:
runs/day
jobs/run
log refs
artifacts/run
proposal activity
audit events/day
runner count
286. Data Growth
Large-growth areas:
job attempts
events
audit
outbox
log metadata
need retention/partition strategy.
287. Partitioning
Do not introduce Postgres partitioning on day one.
Add for:
audit/event scale
very large job history
when measurements justify it.
288. Archival
Old historical metadata can be archived/exported later.
Core transactional correctness should not depend on archive tier.
289. Read Model
If UI queries become expensive:
materialized read models
may be added.
Do not prematurely duplicate all domain data.
290. Read Model Consistency
Read models are derived/eventual.
Critical decisions always use authoritative tables/services.
291. Cache Above Store
In-memory/cache layer may cache:
project config
policy bundle
toolchain metadata
but authoritative mutations still go to store.
292. Cache Invalidation
Use:
version
event
TTL
not indefinite stale caching.
293. Database Is Not General Cache
Do not store:
compiler cache blobs
dependency archives
container layers
inside Postgres.
294. PostgreSQL Large Objects
Avoid unless a very specific need appears.
CAS is the object store abstraction.
295. Binary Columns
Small hashes/IDs/evidence are appropriate.
Large artifacts are not.
296. Storage Architecture Security Invariants
- agents never connect directly to DB;
- UI never connects directly to DB;
- domain crates never import DB drivers;
- DB credentials are secret refs;
- tenant scope is explicit;
- parameterized queries only;
- migrations are integrity checked;
- backups are encrypted/protected operationally;
- restore is tested;
- audit writes are append-oriented.
297. Store Reliability Invariants
- cross-system effects are not claimed exactly-once;
- idempotency keys prevent duplicate command effects;
- optimistic concurrency prevents lost updates;
- leases expire and reconcile;
- outbox prevents lost event publication;
- DB corruption is distinct from not-found;
- state + event are committed atomically where needed;
- long external work is outside DB transaction.
298. Standalone Invariants
- no external DB setup required;
- one local authoritative Stoolap DB;
- local backup is supported;
- schema migrations are automatic when safe;
- distributed-only features are not required for correctness.
299. Distributed Invariants
- PostgreSQL/Neon is shared metadata authority;
- CAS stores bulk immutable bytes;
- Raft does not replace Postgres;
- daemon replicas use compatible schema;
- migrations support rolling upgrades;
- database restore is followed by reconciliation.
300. Implementation Phase 1 — Store API
Implement:
StoreError
EntityVersion
pagination
transaction API
ProjectStore
RunStore
JobStore
RunnerStore
301. Phase 2 — Stoolap
Implement:
local schema
transactions
core stores
migration runner
health
backup
Exit:
standalone Forgeyard core workflows persist locally
302. Phase 3 — PostgreSQL
Implement:
pool
transactions
core schema
run/job/runner stores
source/artifact refs
303. Phase 4 — Conformance Testkit
Run same behavioral tests against:
Stoolap
Postgres
304. Phase 5 — Idempotency / Outbox
Implement:
idempotency table/service
outbox
inbox/dedup where needed
publisher reconciliation
305. Phase 6 — VCS / Change Persistence
Integrate already-designed:
VCS metadata/provenance
Change Proposal/review/check/queue
without changing their domain contracts.
306. Phase 7 — Release / Deployment / Audit
Add:
release
deployment
audit
policy decisions
307. Phase 8 — Migration Hardening
Implement:
checksums
expand-contract
backfill framework
migration lock
N/N-1 tests
308. Phase 9 — Backup / Restore
Implement:
metadata backup
restore
manifest
integrity verification
reconciliation
309. Phase 10 — HA / Performance
Implement/validate:
pool tuning
failover behavior
read replicas if needed
large-scale indexes
queue hot path
scheduler hot path
310. Acceptance Tests
- Same domain service works with Stoolap and Postgres.
- Domain crates do not import DB driver.
- Stale entity version update fails.
- Same idempotency key does not duplicate run creation.
- Job lease acquisition is atomic.
- Expired lease reconciles correctly.
- State transition + event commit atomically.
- Outbox retry does not duplicate consumer effect.
- Tenant A cannot read Tenant B resource.
- CAS object bytes are not stored in metadata tables.
- Artifact metadata points to verified CAS object.
- Failed metadata transaction does not expose artifact.
- Orphan CAS object becomes GC-eligible.
- Postgres migration checksum mismatch fails startup.
- Stoolap migration produces compatible domain behavior.
- N-1 compatible daemon can coexist during expand phase.
- Backup restore returns same critical metadata.
- Restore reconciliation marks stale running jobs correctly.
- Integration queue candidate is revalidated after restore.
- Store health detects schema mismatch.
- Database credential never appears in logs.
- Audit event cannot be casually updated.
- Revision→snapshot mutation is rejected.
- High-volume log bytes remain outside SQL.
- Scheduler can claim work without table-wide scans.
311. Production Readiness Gates
Do not call storage production-ready until:
Stoolap adapter passes conformance
Postgres adapter passes conformance
transaction semantics documented
optimistic concurrency works
idempotency works
outbox/reconciliation works
tenant isolation tested
migrations are checksum-verified
expand-contract tested
backup/restore tested
CAS reference integrity checks exist
health/metrics exist
312. Final Architecture
Domain Services
│
▼
Domain Persistence Interfaces
│
┌─────────────┴─────────────┐
▼ ▼
StoolapStore PostgresStore
│ │
▼ ▼
local metadata DB PostgreSQL / Neon
│ │
└─────────────┬─────────────┘
▼
Metadata State
┌────────────┼────────────┐
▼ ▼ ▼
Runs/Jobs VCS/Change Release/Audit
│ │ │
└────────────┼────────────┘
▼
CAS References
│
▼
CAS
313. Final Architectural Position
The persistence rule is:
business operation
↓
service validates policy/invariants
↓
typed store operation
↓
transaction
↓
metadata mutation
+ durable event/outbox
↓
commit
For large immutable data:
produce bytes
↓
write/verify CAS
↓
transactionally register metadata reference
↓
reconcile/GC orphan objects
Standalone:
ForgeyardStore
↓
Stoolap
Distributed:
ForgeyardStore
↓
PostgreSQL / Neon
The domain does not change.
This is the key guarantee:
Forgeyard can move from a zero-setup local binary to a shared enterprise deployment without rewriting business logic around a different database, while still using database-specific optimizations safely inside adapters.
314. Recommended New-Repository Sequence
After 01-forgeyard-core-domain-foundation.md, implement this subsystem before:
pipeline execution
scheduler
runner
Change Proposal persistence
release state
distributed HA
because all of them depend on stable transactional state semantics.
Recommended implementation order now becomes:
01 Core Domain & Foundation
02 Storage & Metadata
03 CAS & Artifact Data Plane
04 Pipeline IR / Parsing / Planning
05 Run / Job State Machine
06 Scheduler
07 Runner / Agent
08 Sandbox & Executor
09 Transport / QUIC / Protocol
10 Events / Reconciliation
11 Policy / Authorization / Identity
12 Secrets & Trust
13 Supply Chain / SBOM / Provenance / Signing
14 Artifact / Packaging
15 Release
16 Deployment
17 Observability / Health / Doctor
18 API / Axum
19 Dioxus UI
20 Device Lab
21 SCM Provider Integrations
22 HA / Coordination / Raft
23 RBE Interop
24 Plugin / Extension Architecture
25 Operations / Backup / Upgrade / DR
26 Self-Hosting / Bootstrap / Release of Forgeyard
The previously completed language, hermetic, VCS-neutral, and Change Proposal documents plug into these foundations rather than being rewritten.