07 — Forgeyard Runner / Agent System Architecture
Document type: Core Execution Runtime System & Architecture
Project: Forgeyard CI/CD
Subsystem: Runner identity, agent lifecycle, registration, capability discovery, lease handling, workspace preparation, CAS materialization, sandbox/executor handoff, heartbeats, logs, output finalization, local persistence, reconnect/recovery, drain/shutdown, device hosting, and platform-specific execution integration
Architecture style: Thin trusted agent coordinating immutable planned work with strict lease authority, local capability discovery, isolated execution, and resumable control-plane communication
Status: Target production architecture for the new Forgeyard repository
Relationship to prior work: Builds directly on 05-forgeyard-run-job-state-machine.md and 06-forgeyard-scheduler-system-architecture.md. It also consumes the CAS data plane, platform/native capability model, pipeline JobSpecId, and transport/protocol architecture.
1. Purpose
Forgeyard runners execute the actual workload.
That makes them one of the highest-risk and most operationally important components in the system.
The runner/agent must:
register honestly
report capabilities
receive only authorized leased work
fetch exact immutable inputs
create isolated workspace
invoke sandbox/executor
stream logs/status
upload outputs
finalize only current attempt
recover safely from disconnects
The central rule is:
The agent coordinates execution but is never the authority for job state, policy, or scheduling.
A second rule is:
Every active execution is bound to a specific JobId + AttemptId + LeaseId + JobSpecId + AgentSessionId.
A third rule is:
The agent never executes arbitrary daemon text commands outside a validated planned job specification.
2. Architectural Position
Forgeyard Daemon
│
QUIC/Postcard
│
▼
Forgeyard Agent
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Capability CAS Materialize Heartbeat
Discovery │ │
▼ │
Workspace │
│ │
▼ │
Sandbox / Executor │
│ │
▼ │
User Workload │
│ │
▼ │
Output Capture │
│ │
▼ │
CAS │
│ │
└───────┬────────┘
▼
Completion
3. Goals
The agent MUST:
- have stable runner identity;
- have ephemeral agent session identity;
- authenticate to daemon;
- negotiate protocol version;
- report capabilities;
- refresh capabilities;
- report liveness;
- accept/reject leases;
- validate lease/job/spec identity;
- fetch CAS inputs;
- verify inputs;
- materialize workspace safely;
- invoke executor abstraction;
- enforce local resource limits;
- stream logs;
- stream significant phase changes;
- upload outputs;
- verify output digests;
- finalize attempts idempotently;
- support reconnect;
- support drain;
- support graceful shutdown;
- support local state recovery;
- handle stale leases safely;
- isolate workloads from agent internals;
- support Linux/Windows/macOS runners;
- support device-hosting runners;
- support signing-restricted workers via separate composition;
- expose metrics/health;
- remain independent of UI/business-policy logic.
4. Non-Goals
The agent does not:
compile pipeline syntax
decide runner placement
evaluate business authorization
become metadata authority
host public REST API
make release approval decisions
5. Workspace Structure
crates/runner/
├── forgeyard-runner/
├── forgeyard-runner-model/
├── forgeyard-runner-identity/
├── forgeyard-runner-registration/
├── forgeyard-runner-capability/
├── forgeyard-runner-session/
├── forgeyard-runner-lease/
├── forgeyard-runner-workspace/
├── forgeyard-runner-materialize/
├── forgeyard-runner-input/
├── forgeyard-runner-output/
├── forgeyard-runner-log/
├── forgeyard-runner-heartbeat/
├── forgeyard-runner-reconnect/
├── forgeyard-runner-drain/
├── forgeyard-runner-local-state/
├── forgeyard-runner-health/
├── forgeyard-runner-metrics/
└── forgeyard-runner-testkit/
Application binary:
apps/forgeyard-agent/
6. Runner Identity
Stable identity:
#![allow(unused)]
fn main() {
pub struct RunnerId(Ulid);
}Represents one provisioned logical runner.
7. Agent Session Identity
#![allow(unused)]
fn main() {
pub struct AgentSessionId(Ulid);
}Changes every agent process/session startup unless explicit recovery reuses validated session state.
Recommended initial behavior:
new process = new AgentSessionId
8. Why Separate RunnerId and SessionId
Because:
same machine
same runner enrollment
new process
must not silently inherit old leases.
9. Runner Enrollment
Provisioning flow:
operator creates/enrolls runner
↓
runner gets bootstrap credential
↓
agent starts
↓
mTLS/token enrollment
↓
RunnerId assigned/bound
↓
long-term runner credential issued
Exact identity mechanism belongs to trust/identity subsystem.
10. Agent Startup
load config
↓
load runner identity
↓
create AgentSessionId
↓
discover local capabilities
↓
connect daemon
↓
authenticate
↓
negotiate protocol
↓
register
↓
enter active loop
11. Agent Main Loop
Conceptually:
control messages
heartbeats
lease offers
capability refresh
log streams
reconnect
shutdown
12. Agent State
#![allow(unused)]
fn main() {
pub enum AgentState {
Starting,
Connecting,
Registering,
Online,
Draining,
Reconnecting,
ShuttingDown,
Offline,
}
}13. Registration Request
#![allow(unused)]
fn main() {
pub struct RegisterAgent {
pub runner: RunnerId,
pub session: AgentSessionId,
pub protocol: ProtocolSupport,
pub version: AgentVersion,
pub capabilities: RunnerCapabilities,
pub capability_digest: CapabilityDigest,
}
}14. Registration Response
#![allow(unused)]
fn main() {
pub struct RegistrationAccepted {
pub server_time: Timestamp,
pub lease_policy: LeasePolicy,
pub heartbeat_policy: HeartbeatPolicy,
pub protocol: ProtocolVersion,
}
}15. Registration Rejection
Reasons:
authentication failed
runner disabled
protocol incompatible
agent version unsupported
capability schema unsupported
16. Capability Discovery
Agent discovers:
OS
architecture
CPU
memory
disk
GPU
sandbox support
toolchains
SDKs
devices
security capabilities
17. Capability Discovery Crates
Platform adapters provide probes.
Example Linux:
platforms/linux/forgeyard-linux-detect
Runner capability layer composes results.
18. Capability Snapshot
#![allow(unused)]
fn main() {
pub struct RunnerCapabilities {
pub platform: PlatformCapability,
pub resources: ResourceCapacity,
pub toolchains: Vec<ToolchainCapability>,
pub sandbox: Vec<SandboxCapability>,
pub devices: DeviceCapabilitySummary,
pub trust: RunnerTrust,
}
}19. Capability Digest
Canonical digest over capability snapshot.
#![allow(unused)]
fn main() {
pub struct CapabilityDigest(Digest);
}20. Capability Refresh
Refresh triggers:
startup
toolchain install/remove
device attach/detach
GPU state change
major resource availability change
config change
21. Capability Refresh Is Not Every Heartbeat
Avoid heavy repeated reports.
Heartbeat can carry:
capability_digest
and send full snapshot only when changed.
22. Capability Honesty
General capabilities may be locally probed.
Privileged capabilities:
signing
confidential
trusted network
must be provisioned/verified by control plane policy, not self-asserted.
23. Runner Resource Snapshot
#![allow(unused)]
fn main() {
pub struct LocalResourceSnapshot {
pub total: ResourceVector,
pub agent_reserved: ResourceVector,
pub active_allocations: ResourceVector,
pub pressure: ResourcePressure,
}
}24. Resource Pressure
#![allow(unused)]
fn main() {
pub enum ResourcePressure {
Normal,
Cpu,
Memory,
Disk,
Thermal,
Critical,
}
}25. Pressure Response
Agent can reduce advertised allocatable resources or reject new lease.
26. Heartbeat
#![allow(unused)]
fn main() {
pub struct AgentHeartbeat {
pub runner: RunnerId,
pub session: AgentSessionId,
pub sent_at: Timestamp,
pub agent_state: AgentState,
pub active_attempts: Vec<ActiveAttemptSummary>,
pub capability_digest: CapabilityDigest,
pub resource_summary: ResourceSummary,
}
}27. Heartbeat Frequency
Configurable.
Should balance:
failure detection
network overhead
DB/event load
28. Heartbeat Coalescing
Daemon should not persist every field of every heartbeat as append-only event.
Runner liveness is high-frequency operational state.
29. Lease Offer
#![allow(unused)]
fn main() {
pub struct LeaseOffer {
pub lease: JobLease,
pub spec: PlannedJobSpec,
pub spec_id: JobSpecId,
pub input_manifest: CasObjectRef,
}
}30. Agent Lease Validation
Before accepting:
runner/session match
protocol compatible
lease not expired
job spec supported
required capabilities still present
local resource availability sufficient
agent not draining
31. Lease Acceptance
#![allow(unused)]
fn main() {
pub struct AcceptLease {
pub lease: LeaseId,
pub attempt: JobAttemptId,
pub runner: RunnerId,
pub session: AgentSessionId,
pub spec: JobSpecId,
}
}32. Lease Rejection
#![allow(unused)]
fn main() {
pub struct RejectLease {
pub lease: LeaseId,
pub reason: LeaseRejectReason,
}
}33. Rejection Reasons
#![allow(unused)]
fn main() {
pub enum LeaseRejectReason {
Draining,
ResourcePressure,
CapabilityChanged,
DiskInsufficient,
UnsupportedSpec,
LocalPolicy,
InternalPreparationFailure,
}
}34. Reject Before Work
Prefer rejection before mutating workspace/starting workload.
35. Accepted Lease Lifecycle
Accepted
↓
Preparing
↓
Running
↓
UploadingOutputs
↓
Completed
36. Agent Attempt Context
#![allow(unused)]
fn main() {
pub struct AttemptContext {
pub job: JobId,
pub attempt: JobAttemptId,
pub lease: LeaseId,
pub runner: RunnerId,
pub session: AgentSessionId,
pub spec: JobSpecId,
}
}Every execution-side action carries this context.
37. Local Attempt Registry
In memory:
AttemptId -> AttemptRuntime
Guarded against duplicate lease messages.
38. Local Persistence
forgeyard-runner-local-state
Stores minimal recoverable metadata:
active attempt IDs
lease IDs
spec IDs
workspace paths
sandbox IDs
phase
39. Local State Is Not Authority
Daemon remains truth.
Local state only assists recovery/cleanup.
40. Local State Path
Example:
/var/lib/forgeyard-agent/state/
or platform equivalent.
41. Atomic Local State
Write:
temp
fsync
rename
where durability needed.
42. Workspace Root
Separate from CAS:
/var/lib/forgeyard-agent/workspaces/
43. Workspace Per Attempt
workspaces/<attempt-id>/
or randomized internal path.
44. Workspace Lifecycle
create temp
↓
materialize inputs
↓
configure sandbox
↓
run
↓
capture outputs
↓
cleanup
45. Workspace Isolation
No sharing mutable workspace between unrelated attempts.
46. Input Materialization
input manifest
↓
batch local CAS missing check
↓
fetch
↓
digest verify
↓
tree materialize
↓
read-only source/input
47. Source Path
Typically:
/work/source
inside sandbox.
48. Build Path
Writable:
/work/build
49. Output Path
Declared output roots only.
50. Toolchain Paths
Immutable toolchain store mounted read-only where possible.
51. CAS Safety
Never let workload modify shared CAS file.
52. Reflink/Copy-on-Write
Allowed optimization.
53. Hardlink
Only if strict read-only guarantees prevent mutation.
Default avoid for writable inputs.
54. Materialization Failure
Classify:
InputFetch
CorruptInput
DiskInsufficient
PathIncompatible
55. Corrupt Input
Agent must not continue.
Report integrity error.
56. Missing Input
Try configured fallback tiers.
If unavailable:
attempt preparation failure
57. Sandbox Handoff
Agent constructs:
#![allow(unused)]
fn main() {
pub struct ExecutionRequest {
pub context: AttemptContext,
pub workspace: PreparedWorkspace,
pub spec: PlannedJobSpec,
pub resources: ResourceReservation,
pub environment: RuntimeEnvironment,
}
}passes to Executor.
58. Agent Does Not Shell-Execute Directly
All workload execution goes through executor abstraction.
59. Executor Trait Boundary
#![allow(unused)]
fn main() {
#[async_trait]
pub trait Executor {
async fn execute(
&self,
request: ExecutionRequest,
) -> Result<ExecutionHandle, ExecutorError>;
}
}Detailed executor architecture comes next.
60. Secret Resolution
Agent requests/receives late-bound secret material through secret provider/control plane.
Never store in CAS.
61. Secret Lifetime
resolve
↓
inject
↓
use
↓
zeroize/remove
62. Secret Persistence
Do not write secrets to local runner state.
63. Environment Construction
Combine:
validated literal env
system env
late secrets
runtime paths
64. Host Environment Leakage
Strict mode starts from sanitized environment.
No automatic inheritance of:
HOME
SSH_AUTH_SOCK
AWS credentials
developer PATH
unless explicitly allowed.
65. Preparation Phase Event
Agent sends:
AttemptPreparing
after lease accepted and before substantial prep.
66. Running Phase Event
Send only after executor actually starts workload.
67. Output Phase Event
Send when workload exits and output capture starts.
68. Significant Phase Idempotency
Duplicate phase messages safe.
69. Logs
forgeyard-runner-log
Captures:
stdout
stderr
structured step events
agent diagnostics
70. Log Sequence
Per attempt:
#![allow(unused)]
fn main() {
pub struct LogSeq(u64);
}Monotonic.
71. Log Frame
#![allow(unused)]
fn main() {
pub struct LogFrame {
pub attempt: JobAttemptId,
pub seq: LogSeq,
pub stream: LogStreamKind,
pub bytes: BoundedBytes,
pub timestamp: Timestamp,
}
}72. Log Streams
#![allow(unused)]
fn main() {
pub enum LogStreamKind {
Stdout,
Stderr,
Agent,
Structured,
}
}73. Log Backpressure
If daemon/network slow:
bounded memory buffer
local spool
CAS chunk flush
74. No Infinite RAM Buffer
Never buffer unbounded logs in memory.
75. Local Log Spool
Optional:
workspaces/<attempt>/logs/
or dedicated spool area.
76. Log Loss Policy
For production builds:
best effort live stream
durable chunk spool
so temporary disconnect does not lose all logs.
77. Log Redaction
Secret redaction before durable upload where possible.
78. Redaction Limit
No perfect generic secret detection guarantee.
Treat logs as sensitive.
79. Log Reconnect
On reconnect, agent reports last locally available sequence.
Daemon requests missing range.
80. Output Capture
After successful executor exit:
declared outputs only
are captured.
81. Output Traversal
Validate:
path under workspace
symlinks safe
size limits
file count
82. Output Tree
Canonicalize into CAS tree/manifest.
83. Output Hashing
BLAKE3 internal.
84. Output Upload
Tiered CAS upload according to durability class.
85. Output Metadata
Agent returns CAS refs.
Daemon registers artifact semantics.
86. Agent Does Not Create Release Metadata
Agent only produces execution outputs/evidence.
87. Output Upload Retry
Can retry without rerunning workload while workspace retained.
88. Workspace Retention During Finalize
Keep until:
completion acknowledged
or finalization TTL expires.
89. Completion Request
#![allow(unused)]
fn main() {
pub struct CompleteAttemptRequest {
pub context: AttemptContext,
pub result: AttemptExecutionResult,
pub outputs: Vec<CasObjectRef>,
pub logs: Option<LogStreamRef>,
pub message_id: MessageId,
}
}90. Completion Ack
Daemon returns:
#![allow(unused)]
fn main() {
pub enum CompletionAck {
Accepted,
DuplicateAccepted,
StaleRejected,
InvalidRejected,
}
}91. Stale Completion
Agent MUST stop retrying as authoritative after:
StaleRejected
May keep local diagnostics briefly.
92. Duplicate Accepted
Safe to cleanup.
93. Unknown Completion Result
Network lost before ack:
retry same MessageId.
94. Completion Idempotency
Agent must persist completion message ID until acknowledged if local recovery desired.
95. Cleanup
After accepted terminal result:
remove secrets
destroy sandbox
delete workspace
release local resources
trim spool
96. Cleanup Failure
Report health issue.
Do not change already accepted Job result.
97. Cleanup Reconciler
Periodic scan removes abandoned workspaces.
98. Workspace TTL
Unreferenced old workspaces deleted after grace.
99. Debug Retention
Policy may retain failed workspace locally temporarily.
Disabled by default in sensitive environments.
100. Drain
#![allow(unused)]
fn main() {
pub enum DrainMode {
Graceful,
CancelActive,
}
}101. Graceful Drain
stop accepting new leases
finish active attempts
disconnect when empty
102. Cancel Drain
Requests cancellation for active attempts through control plane.
Agent should not independently mark jobs cancelled.
103. Shutdown
Signal handling:
SIGTERM
Ctrl-C
Windows service stop
moves agent to draining/shutdown.
104. Shutdown Timeout
After grace:
hard terminate local workload
according to configured policy.
105. Agent Restart
Initial recommended behavior:
new session
report old local attempt records
daemon decides stale/lost
cleanup
106. Process Reattachment
Advanced future capability.
Not required initially.
107. Recovery Handshake
#![allow(unused)]
fn main() {
pub struct RecoverAttempts {
pub runner: RunnerId,
pub new_session: AgentSessionId,
pub previous: Vec<RecoveredAttemptSummary>,
}
}108. Daemon Recovery Decision
#![allow(unused)]
fn main() {
pub enum RecoveryDecision {
AbandonAndCleanup,
ContinueSupported,
Stale,
}
}Initial implementation:
AbandonAndCleanup
for restarted process attempts.
109. Reconnect Without Restart
Same AgentSessionId.
Active attempts can continue if lease valid.
110. Connection Loss
Agent enters:
Reconnecting
111. During Short Disconnect
Workload may continue according to lease connectivity policy.
112. Lease Connectivity Policy
#![allow(unused)]
fn main() {
pub enum LeaseConnectivityPolicy {
Continuous,
Grace(Duration),
OfflineAuthorized(Duration),
}
}113. Default Distributed Policy
Short grace.
114. Offline Edge Mode
Can allow longer disconnected execution if explicitly authorized.
115. Lease Expiry Locally
Agent must stop workload when hard lease deadline exceeded if unable to renew.
116. Runner Monotonic Timer
Use local monotonic time for enforcement.
117. Clock Skew
Daemon absolute expiry converted to safe local monotonic deadline at receipt.
118. Reconnect Resync
Agent sends:
active attempts
phase
last log seq
lease ID
119. Daemon Authority Response
Continue
Cancel
Stale
120. Continue
Agent resumes reporting and renews lease.
121. Cancel
Agent terminates workload and reports cancellation.
122. Stale
Agent terminates workload immediately, does not finalize as authoritative.
123. Runner Health
#![allow(unused)]
fn main() {
pub struct RunnerHealth {
pub status: RunnerHealthStatus,
pub checks: Vec<RunnerHealthCheck>,
}
}124. Health Status
#![allow(unused)]
fn main() {
pub enum RunnerHealthStatus {
Healthy,
Degraded,
Unhealthy,
}
}125. Health Checks
disk space
CAS access
sandbox availability
executor probe
toolchain store
device daemon
clock sanity
credential validity
126. Unhealthy Runner
Agent remains connected but scheduler should receive no new leases.
127. Degraded Runner
May accept limited jobs depending reason/policy.
128. Agent Doctor
forgeyard-agent doctor
or:
forgeyard runner doctor
Checks all required local dependencies.
129. Runner CLI
On host:
forgeyard-agent status
forgeyard-agent doctor
forgeyard-agent capabilities
forgeyard-agent drain
forgeyard-agent resume
forgeyard-agent cleanup
130. Remote Admin
Control-plane CLI:
forgeyard runner list
forgeyard runner show
forgeyard runner drain
forgeyard runner disable
131. Capability Output
Human-readable:
Linux x86_64
32 CPU
64 GiB
Rust stable ...
Docker/container executor ...
Android devices ...
132. Local Configuration
config/default/agent.ron
Example:
(
data_dir: "/var/lib/forgeyard-agent",
max_parallel_jobs: 4,
workspace_retention: "30m",
reconnect: (
max_backoff: "30s",
),
)
133. Runner Labels
Optional.
Used for:
site
ownership
operator hints
Not privileged trust.
134. Runner Pool Binding
Agent can be assigned to pool by server/admin.
Do not trust self-selected restricted pool.
135. Trust Boundary
Untrusted workload must never access:
agent credentials
daemon mTLS key
runner enrollment token
other job workspace
host secrets
136. Agent Credential Isolation
Keep credentials outside workload-visible mount namespace.
137. IPC Boundary
Prefer no direct workload-to-agent IPC.
138. Host Socket Leakage
Do not mount:
Docker socket
SSH agent
system DBus
unless explicitly needed and policy-approved.
139. Privilege
Agent may require host privileges for sandbox setup.
Workload should run with far lower privileges.
140. Privilege Separation
Potential future:
unprivileged agent
+
small privileged helper
for Linux namespace/cgroup setup.
141. Initial Linux Recommendation
Use narrowly scoped privileged operations and isolate helper logic in platform sandbox crate.
142. Windows Runner
Platform adapter handles:
Job Objects
ACLs
process tokens
Windows SDK
143. macOS Runner
Platform adapter handles:
sandbox/process controls
Xcode
simulators
signing only if allowed
144. Agent Core Remains Cross-Platform
No large platform-specific cfg forest in runner model/service.
145. Platform Adapter Trait
#![allow(unused)]
fn main() {
pub trait RunnerPlatform {
fn discover_capabilities(&self) -> Result<PlatformCapabilities, PlatformError>;
fn prepare_workspace(&self, ...) -> ...;
}
}146. Device Hosting
Runner can host physical devices.
Device subsystem reports:
attached
available
leased
health
147. Device Isolation
One physical device lease at a time unless device supports safe parallelism.
148. Device Runner Workflow
job lease
+
device lease
↓
prepare
install/flash
run tests
collect logs
cleanup/reset
149. Device Cleanup
Mandatory before device returns to pool.
150. Device Failure
Classify separately from job workload.
151. Signing Worker Distinction
Do not simply enable signing capability on general-purpose agent.
Use:
apps/forgeyard-signing-worker
with restricted code path.
152. Runner Agent vs Signing Worker
Runner:
arbitrary untrusted build code
Signing worker:
no arbitrary build code
restricted signing requests only
153. Confidential Runner
Requires special executor/trust integration.
Agent core only consumes capability/policy.
154. Toolchain Discovery
Agent should prefer immutable managed toolchain registry over scanning arbitrary PATH.
155. Host Toolchains
Can be reported in impure/dev mode.
Marked distinctly.
156. Managed Toolchain Capability
#![allow(unused)]
fn main() {
pub struct ToolchainCapability {
pub id: ToolchainId,
pub source: ToolchainSource,
pub verified: bool,
}
}157. Toolchain Warm Cache
Agent can report a bounded set of warm toolchains.
Scheduler uses as soft preference.
158. Auto-Install Toolchain
Agent may materialize required toolchain from CAS.
Does not need preinstalled capability if runner supports generic platform + toolchain materializer.
159. Capability Semantics
Distinguish:
can materialize toolchain
from:
toolchain already warm
160. Input Prefetch
Agent may prefetch leased job inputs before full sandbox creation.
161. Speculative Prefetch
Only if scheduler/control plane explicitly requests.
Do not waste bandwidth by guessing excessively.
162. Local CAS
Agent has L1 CAS.
163. CAS Eviction
Independent local cache policy.
Never delete active attempt required objects while in use.
164. CAS Pin During Attempt
Pin/local reference count required input closure until attempt cleanup.
165. Disk Quota Per Attempt
Prevent one job from exhausting host.
166. Workspace Size Accounting
Track:
input materialized size
working data
outputs
logs
167. Resource Enforcement
Agent delegates hard limits to sandbox/executor.
168. Soft Usage Monitoring
Agent can monitor actual:
CPU
memory
disk
for metrics.
169. OOM
Sandbox/executor reports:
OutOfMemory
mapped to failure class.
170. Disk Full
Likewise.
171. Timeout Enforcement
Agent has local timers for:
prepare
run
upload
172. Cancellation Token
Per attempt local token propagated into executor.
173. Graceful Process Termination
Executor:
soft signal
grace period
hard kill
platform-specific.
174. Child Process Cleanup
Mandatory.
No orphan child processes after attempt.
175. Background Daemons Started by Job
Sandbox/process group cleanup kills them.
176. Workspace Escape Detection
Sandbox should prevent.
Agent cleanup also validates no unexpected mounts/processes remain.
177. Output Capture After Failure
Policy may capture diagnostics even for failed jobs.
178. Failure Artifacts
Examples:
test report
core dump
screenshots
crash logs
subject to size/security policy.
179. Core Dumps
Disabled by default or restricted due to secrets.
180. Debug Artifacts
Can be marked sensitive.
181. Agent Internal Logs
Separate from job logs.
182. Agent Log Fields
runner
session
attempt
lease
phase
No secret values.
183. Structured Events
Use stable event types for:
lease accepted
workspace prepared
executor started
upload completed
cleanup failed
184. Metrics
agent_active_jobs
agent_cpu_used
agent_memory_used
agent_disk_free
agent_lease_accept
agent_lease_reject
agent_reconnects
agent_input_fetch_bytes
agent_output_upload_bytes
agent_workspace_cleanup_failures
agent_log_spool_bytes
185. Attempt Metrics
prepare_duration
execution_duration
upload_duration
cleanup_duration
186. High Cardinality
Do not label metrics with JobId.
Tracing carries IDs.
187. Health Endpoint
Local-only/admin:
/health
if an HTTP endpoint is used.
Could instead expose IPC/CLI.
188. Public Exposure
Do not expose runner admin endpoint publicly by default.
189. Reconnect Backoff
#![allow(unused)]
fn main() {
pub struct ReconnectPolicy {
pub initial: Duration,
pub max: Duration,
pub jitter: JitterPolicy,
}
}190. Reconnect Behavior
Retry forever unless:
credential revoked
protocol permanently incompatible
runner disabled
191. Credential Rotation
Agent supports server-issued new credential/cert.
192. Certificate Expiry
Warn before expiry.
193. Control-Plane Discovery
Agent config may contain:
daemon endpoint
cluster endpoint
HA discovery details later.
194. Multiple Daemon Endpoints
Agent can fail over among cluster endpoints.
195. Connection Stickiness
Keep one active control stream; reconnect elsewhere on failure.
196. Duplicate Connections
Daemon prevents one same session from being simultaneously authoritative on two channels, or treats latest authenticated connection as active.
197. Session Fencing
Registration can return:
session epoch/token
if needed.
198. Initial Recommendation
Use unique AgentSessionId and server-side active-session record.
199. Runner Disable
Admin sets runner disabled.
Agent connection may remain only long enough to cancel/drain.
200. Runner Delete
Logical deletion after active leases cleared.
201. Runner Re-enrollment
Creates/reuses RunnerId according to admin policy.
202. Host Reimage
Recommended:
new enrollment credential/session
Stable RunnerId optional if inventory identity preserved.
203. Runner Tags vs Identity
Do not derive identity from hostname.
Hostname can change/collide.
204. Machine Fingerprint
Can be diagnostic, not sole security identity.
205. Runner Version
#![allow(unused)]
fn main() {
pub struct AgentVersion {
pub forgeyard_version: SemVersion,
pub protocol_support: ProtocolRange,
}
}206. N/N-1
Agent and daemon support rolling compatibility as defined in protocol architecture.
207. Unsupported Version
Agent does not run jobs until upgraded.
208. Self-Update
Potential future.
Do not make runner auto-update mandatory initially.
209. Self-Update Security
Would require signed release verification and rollback.
Separate architecture later if needed.
210. Agent Installation
Platform packages:
systemd service
Windows service
launchd
211. Linux Service
Runs dedicated user where possible.
212. Windows Service
Dedicated service account.
213. macOS Service
Dedicated daemon account/process.
214. Permissions
Grant only sandbox/device capabilities needed.
215. Rootless Mode
Support where sandbox capabilities allow.
216. Privileged Mode
Explicitly surfaced in doctor/UI.
217. Runner Local Policy
Agent may enforce stricter local limits than central job request.
Example:
never allow host networking
218. Local Policy Cannot Weaken Central Policy
It may only tighten.
219. Lease Validation Against Local Policy
Reject lease if local stricter rule conflicts.
220. Agent Config Reload
Safe fields can hot-reload:
log level
capacity limits
drain
Sensitive structural changes may require restart.
221. Config Reload Audit
Server/admin change audited if centrally managed.
222. Runner Capacity Override
Admin can reserve:
2 CPUs for host
8 GiB RAM
223. Effective Allocatable
physical
- agent reserve
- admin reserve
- active reservations
224. Max Parallel Jobs
Hard local guard even if scheduler bug.
225. Defense in Depth
Agent checks resource reservation and capability again before execution.
226. Scheduler Bug Safety
If impossible lease received:
reject
not blindly execute.
227. JobSpec Validation
Agent validates schema/protocol + signatures/digest if applicable.
228. JobSpec Immutable
JobSpecId digest must match received bytes.
229. JobSpec Contents
Includes:
steps/actions
resource limits
input refs
output declarations
network policy
sandbox requirements
timeouts
secret refs
230. No Dynamic Command Mutation
Daemon cannot alter command mid-attempt without new spec/lease.
231. Runtime Control Messages
Allowed:
cancel
lease renewal
log backfill request
Not:
replace shell command
232. Attempt Control Channel
Per attempt logical stream optional over shared QUIC connection.
233. Multiplexing
QUIC streams can separate:
control
logs
CAS transfer
attempt events
234. Agent Transport Independence
Runner core should depend on transport trait, not Quinn-specific internals everywhere.
235. Transport Trait
#![allow(unused)]
fn main() {
#[async_trait]
pub trait AgentControlTransport {
async fn send_event(&self, ...);
async fn recv_command(&self, ...);
}
}236. Internal Backpressure
Bound control channel queues.
237. Priority
Cancellation/lease renew messages higher priority than verbose logs.
238. Log Channel Failure
Should not block cancellation/control.
239. CAS Transfer Channel Failure
Can retry independently.
240. Agent Event Ordering
Per attempt significant phase events should maintain logical order.
241. Sequence Number
Optional:
#![allow(unused)]
fn main() {
pub struct AttemptEventSeq(u64);
}242. Duplicate Event
Daemon deduplicates.
243. Local Event Journal
Optional minimal journal for reconnect.
244. Initial Recommendation
Persist:
completion message
active attempt metadata
log spool
not every phase event.
245. Workspace Cleanup on Startup
Scan old directories.
Compare with local attempt state.
Cleanup unknown stale workspaces after grace.
246. Active Sandbox Detection
If process still running from previous crash:
initial implementation:
terminate + cleanup
unless reattachment supported.
247. Safety First
Never leave unknown orphan workload running after agent restart.
248. Host Reboot
All active attempts likely lost.
On startup report/reconcile.
249. Resume Long Download
CAS subsystem can resume partial objects.
250. Resume Upload
Likewise.
251. Attempt Preparation Checkpoint
Could reuse fetched inputs after retry.
Local cache handles naturally.
252. Workspace Reuse
Do not reuse previous attempt mutable workspace by default.
New attempt gets fresh workspace.
253. Why Fresh Retry Workspace
Avoid contaminated state.
254. Cache Reuse
Reuse through CAS/action cache only.
255. Testkit
forgeyard-runner-testkit/src/
├── lib.rs
├── fake_daemon.rs
├── fake_transport.rs
├── fake_executor.rs
├── fake_cas.rs
├── capability.rs
├── workspace.rs
├── lease.rs
└── assertions.rs
256. Unit Tests
Test:
state
lease validation
capability refresh
workspace paths
cleanup
reconnect backoff
257. Integration Tests
- register agent;
- receive lease;
- fetch input;
- run fake executor;
- upload output;
- complete.
258. Duplicate Lease Test
Same LeaseId sent twice.
One local attempt only.
259. Stale Lease Test
Expired/stale lease rejected.
260. Capability Change Test
Lease requires GPU; GPU removed before accept -> reject.
261. Disconnect Test
Disconnect during Running, reconnect, resync, continue.
262. Stale After Reconnect Test
Daemon says Stale -> terminate.
263. Log Backpressure Test
Network stalled; memory remains bounded; spool used.
264. Output Retry Test
CAS temporary failure; output upload retried without rerun.
265. Completion Ack Loss Test
Completion committed but ack lost; same message retried, duplicate accepted.
266. Agent Crash Test
Crash during Running; restart cleans orphan and control plane marks lost.
267. Drain Test
Draining agent rejects new lease and completes active attempt.
268. Shutdown Test
Graceful + forced timeout.
269. Security Tests
- workload cannot read agent credential;
- workload cannot access another workspace;
- absolute path escape rejected;
- host env secrets absent;
- stale lease cannot execute;
- malformed JobSpec rejected;
- unauthorized signing capability impossible on general agent.
270. Cross-Platform Tests
Linux, Windows, macOS agent behavior through same core contracts.
271. Device Tests
Attach/detach/lease/reset.
272. Performance Tests
Measure:
registration
input materialization
log throughput
output hashing
concurrent attempt overhead
273. Scale Tests
One agent with many concurrent small jobs.
274. Resource Leak Test
After 1000 attempts:
no leaked workspaces
no leaked child processes
no leaked resource reservations
275. Fuzzing
Targets:
JobSpec decoder
control messages
local state recovery
log frames
276. Failure Injection
disk full
CAS corrupt
network partition
executor crash
sandbox setup fail
credential expiry
277. Observability Acceptance
Operator can answer:
why runner is not receiving jobs
which attempts are active
what capability changed
why lease rejected
how much disk/CAS used
278. Implementation Phase 1 — Identity / Registration
Implement:
RunnerId
AgentSessionId
registration
heartbeat
basic health
279. Phase 2 — Capability Discovery
Implement platform/resource/toolchain capability reporting.
280. Phase 3 — Lease Loop
Implement:
receive
validate
accept/reject
dedupe
281. Phase 4 — Workspace / CAS
Implement:
per-attempt workspace
input fetch/materialize
local CAS pinning
282. Phase 5 — Executor Handoff
Integrate generic executor trait.
283. Phase 6 — Logs / Phase Events
Implement bounded streaming/spool.
284. Phase 7 — Output / Completion
Implement output capture, CAS upload, idempotent completion.
285. Phase 8 — Drain / Shutdown
Implement operator lifecycle.
286. Phase 9 — Reconnect / Recovery
Implement same-session reconnect and restart cleanup.
287. Phase 10 — Devices / Advanced Platforms
Integrate device-hosting and platform-specific capabilities.
288. Phase 11 — Hardening
Security, failure injection, high concurrency, protocol compatibility.
289. Acceptance Tests
- Agent registers with stable RunnerId and new SessionId.
- Protocol incompatibility blocks Online state.
- Capability snapshot is deterministic.
- Capability change triggers refresh.
- Draining agent rejects new leases.
- Expired lease is rejected.
- Wrong RunnerId/SessionId lease is rejected.
- Duplicate LeaseId creates one local execution.
- Input CAS bytes are digest verified.
- Corrupt input prevents execution.
- Workspace is unique per attempt.
- Workload cannot mutate shared CAS.
- Host environment is sanitized.
- Executor receives exact JobSpec.
- Agent cannot alter planned command.
- Phase transitions are reported idempotently.
- Logs remain bounded under network backpressure.
- Reconnect can backfill logs.
- Output capture is limited to declared roots.
- Output digest is verified.
- Completion includes Job/Attempt/Lease/Spec/Session.
- Lost completion ACK is safely retried.
- StaleRejected stops authoritative retry.
- Cleanup happens after accepted completion.
- Agent crash leaves no permanently trusted state.
- Restart creates new session and reconciles old attempts.
- Graceful drain completes active work.
- Forced shutdown terminates child processes.
- General runner cannot self-assert signing trust.
- Device hosting follows separate device lease.
- Linux/Windows/macOS share same core agent semantics.
- Same agent core works in standalone and distributed compositions.
- UI has no direct agent control authority.
- Metrics expose pressure/rejections/reconnects.
- Forgeyard self-hosting runners can execute Forgeyard jobs through this agent.
290. Production Readiness Gates
Do not call runner/agent production-ready until:
registration/auth stable
capability discovery verified
lease validation correct
input integrity enforced
workspace isolation stable
executor boundary enforced
bounded logs implemented
output commit idempotent
reconnect tested
drain/shutdown tested
restart cleanup tested
security tests pass
291. Architectural Invariants
- RunnerId is stable provisioned identity.
- AgentSessionId is ephemeral process/session identity.
- Lease is bound to runner + session.
- Agent never becomes job-state authority.
- Agent never becomes scheduler authority.
- Agent executes only validated immutable JobSpec.
- JobSpecId must match received spec.
- Duplicate lease delivery is safe.
- Stale lease never executes.
- Shared CAS is immutable to workload.
- Workspace is unique per attempt.
- Retry gets fresh workspace.
- Host environment is sanitized.
- Secrets are late-bound and never persisted in normal local state.
- Logs are bounded and recoverable.
- Output capture follows declarations.
- CAS upload alone does not finalize job.
- Completion is idempotent.
- Stale completion is rejected by control plane.
- Agent cleanup does not rewrite job outcome.
- Restart never silently inherits old authority.
- Reconnect resynchronizes active attempts.
- Hard lease expiry eventually stops disconnected work.
- General runners cannot self-grant privileged trust.
- Platform-specific behavior lives in adapters.
- Signing worker remains separate from general runner.
- Device control uses separate device lease.
- In-memory state is not sole authority.
- Local persisted state is recovery aid only.
- Forgeyard should dogfood its own agent runtime.
292. Final Target Architecture
Forgeyard Daemon
│
Authenticated QUIC
│
▼
Forgeyard Agent
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Registration Capability Heartbeat
│
▼
Lease Offer
│
validate/dedupe
│
▼
Accept
│
▼
AttemptContext
│
▼
Fetch CAS Inputs
│
▼
Prepare Workspace
│
▼
Sandbox / Executor
│
▼
Workload
│
logs / phase events
│
▼
Capture Outputs
│
▼
CAS
│
▼
Completion Request
│
▼
Daemon validates
│
Accepted / Stale
│
▼
Cleanup
293. Final Architectural Position
Agent authority:
RunnerId
+
AgentSessionId
+
LeaseId
+
AttemptId
+
JobSpecId
Execution:
validated lease
↓
verified CAS inputs
↓
fresh workspace
↓
sandbox/executor
↓
bounded logs
↓
declared outputs
↓
verified CAS upload
Finalization:
same authority tuple
+
result
+
outputs
↓
daemon
↓
accepted / duplicate / stale
The key guarantee is:
A Forgeyard agent is powerful enough to execute untrusted workloads, but deliberately too weak to invent work, choose its own privileges, alter job semantics, or declare itself successful without the control plane validating its exact lease, attempt, session, specification, and outputs.
294. New-Repository Sequence
01 Core Domain & Foundation
02 Storage & Metadata
03 CAS & Artifact Data Plane
04 Pipeline IR / Parsing / Planning
05 Run / Job State Machine
06 Scheduler
07 Runner / Agent
08 Sandbox & Executor
09 Transport / QUIC / Protocol
10 Events / Reconciliation
11 Policy / Authorization / Identity
12 Secrets & Trust
13 Supply Chain / SBOM / Provenance / Signing
14 Packaging
15 Release
16 Deployment
17 Observability / Health / Doctor
18 API / Axum
19 Dioxus UI
20 Device Lab
21 SCM Provider Integrations
22 HA / Coordination / Raft
23 RBE Interop
24 Plugin / Extension Architecture
25 Operations / Backup / Upgrade / DR
26 Self-Hosting / Bootstrap / Release of Forgeyard