06 — Forgeyard Scheduler System Architecture
Document type: Core Execution Orchestration System & Architecture
Project: Forgeyard CI/CD
Subsystem: Job eligibility, placement, runner selection, capability matching, resource accounting, locality scoring, fairness, queueing, priorities, lease issuance, backpressure, preemption policy, scheduling recovery, and distributed-safe orchestration
Architecture style: Two-phase scheduling — hard eligibility filtering followed by soft scoring — with persisted lease authority and reconciliation
Status: Target production architecture for the new Forgeyard repository
Relationship to prior work: Builds directly on 05-forgeyard-run-job-state-machine.md, consuming Eligible jobs and creating authoritative attempts/leases. It also integrates with the CAS data plane, pipeline capability planning, platform/device capability models, and storage layer.
1. Purpose
Forgeyard needs a scheduler that can place work correctly across:
local runner
distributed Linux runners
Windows runners
macOS runners
Android device pools
Apple device pools
GPU runners
high-memory runners
special toolchain runners
trusted signing workers
confidential execution workers
The scheduler must decide:
where should this eligible job run?
without becoming the authority for:
job state
attempt state
lease truth
The central rule is:
The scheduler decides placement; the Run/Job state machine and store decide authority.
A second rule:
Scheduling is always hard filtering first, soft scoring second.
A third rule:
A runner is eligible only if every required capability can be satisfied. Locality, load, queue time, cache warmth, and cost influence ranking only after correctness constraints are met.
2. Architectural Position
Eligible Jobs
│
▼
Scheduler Ingress
│
▼
Hard Requirement Match
│
▼
Eligible Runner Set
│
▼
Soft Scoring
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Locality Headroom Fairness
│ │ │
└──────────────┼──────────────┘
▼
PlacementDecision
│
▼
Attempt + Lease Transaction
│
▼
Runner
3. Goals
The scheduler MUST:
- consume only jobs in
Eligible; - never mutate job state directly outside store/state APIs;
- match all required capabilities;
- support heterogeneous runner pools;
- support OS/architecture constraints;
- support toolchain/SDK constraints;
- support resource constraints;
- support GPU/device/special hardware;
- support trust/security requirements;
- support data locality;
- support cache locality;
- support queue fairness;
- support tenant/project fairness;
- support job priorities;
- support backpressure;
- support retry-aware placement;
- support anti-affinity/reliability heuristics;
- support runner drain;
- support capacity reservations;
- support preemption where explicitly allowed;
- support standalone mode;
- support distributed mode;
- support HA-safe multiple daemon instances;
- support deterministic explainability;
- expose scheduler metrics;
- tolerate runner churn;
- tolerate stale capability reports;
- avoid scheduling correctness based on predictions alone;
- integrate with device leases;
- integrate with CAS locality without making CAS locality mandatory.
4. Non-Goals
The scheduler does not:
execute jobs
validate pipeline syntax
resolve secrets
perform sandbox setup
store artifact bytes
decide business authorization
It uses upstream validated data.
5. Workspace Structure
crates/scheduler/
├── forgeyard-scheduler/
├── forgeyard-scheduler-model/
├── forgeyard-scheduler-eligibility/
├── forgeyard-scheduler-placement/
├── forgeyard-scheduler-score/
├── forgeyard-scheduler-resource/
├── forgeyard-scheduler-fairness/
├── forgeyard-scheduler-priority/
├── forgeyard-scheduler-queue/
├── forgeyard-scheduler-backpressure/
├── forgeyard-scheduler-preemption/
├── forgeyard-scheduler-locality/
├── forgeyard-scheduler-retry/
├── forgeyard-scheduler-leasing/
├── forgeyard-scheduler-reconcile/
├── forgeyard-scheduler-metrics/
└── forgeyard-scheduler-testkit/
6. Scheduler Input
The scheduler should consume an immutable scheduling view:
#![allow(unused)]
fn main() {
pub struct SchedulingJob {
pub job_id: JobId,
pub run_id: RunId,
pub spec: JobSpecId,
pub requirements: CapabilityRequirementSet,
pub resources: ResourceRequest,
pub priority: JobPriority,
pub locality: LocalityHints,
pub retry_context: RetryContext,
}
}7. Runner Scheduling View
#![allow(unused)]
fn main() {
pub struct SchedulingRunner {
pub runner_id: RunnerId,
pub session_id: AgentSessionId,
pub state: RunnerSchedulingState,
pub capabilities: RunnerCapabilities,
pub resources: RunnerResources,
pub locality: RunnerLocality,
pub trust: RunnerTrust,
pub recent_health: RunnerHealthSummary,
}
}8. Runner Scheduling States
#![allow(unused)]
fn main() {
pub enum RunnerSchedulingState {
Online,
Draining,
Saturated,
Unhealthy,
Offline,
}
}Only:
Online
is normally eligible for new work.
9. Hard Eligibility
Hard eligibility answers:
can this runner execute this job correctly?
This is a boolean decision.
10. Hard Requirement Categories
OS
architecture
CPU features
memory
disk
GPU
toolchain
SDK
runtime
sandbox
device
trust level
network mode
signing capability
confidential execution
11. Capability Match
#![allow(unused)]
fn main() {
pub trait CapabilityMatcher {
fn matches(
&self,
requirement: &CapabilityRequirement,
runner: &RunnerCapabilities,
) -> CapabilityMatch;
}
}12. Capability Match Result
#![allow(unused)]
fn main() {
pub enum CapabilityMatch {
Satisfied,
Unsatisfied(CapabilityMismatch),
Unknown(CapabilityUnknown),
}
}Unknown must not be treated as satisfied for hard requirements.
13. Versioned Capabilities
Example:
Xcode >= 18.0
Android NDK r28
Rust toolchain digest X
Windows SDK version Y
GPU compute capability >= Z
Never reduce to:
"mac"
"android"
"gpu"
14. Platform Matching
Examples:
linux/x86_64
windows/x86_64
macos/aarch64
android/device
wasm/runtime
15. Architecture Matching
Use typed architecture:
x86_64
aarch64
riscv64
wasm32
No freeform strings in core model.
16. CPU Feature Requirement
Example:
AVX2
AES-NI
NEON
Only required if build/test semantics genuinely depend on it.
17. Resource Eligibility
Runner must have enough allocatable resources.
requested CPU <= available CPU
requested memory <= available memory
requested disk <= available disk
GPU request satisfiable
18. Reservation vs Physical Capacity
Runner reports:
physical capacity
reserved capacity
allocatable capacity
Scheduler uses allocatable.
19. Resource Model
#![allow(unused)]
fn main() {
pub struct RunnerResources {
pub total: ResourceVector,
pub reserved: ResourceVector,
pub allocated: ResourceVector,
pub allocatable: ResourceVector,
}
}20. Resource Vector
#![allow(unused)]
fn main() {
pub struct ResourceVector {
pub cpu: CpuUnits,
pub memory: ByteSize,
pub disk: ByteSize,
pub gpu: Vec<GpuResource>,
}
}21. CPU Units
Prefer fractional logical CPU units if needed.
Avoid raw floats.
22. Memory Headroom
Do not schedule exactly to physical limit by default.
Reserve:
OS
agent
CAS
sandbox overhead
23. Disk Headroom
Input/output temp space must be considered.
24. Dynamic Resource Governor
Scheduler may reserve conservative capacity based on:
historical peak
job declarations
platform overhead
Optimization only if bounded by declared hard limits.
25. Soft Scoring
After hard filtering:
which eligible runner is best?
26. Score Model
#![allow(unused)]
fn main() {
pub struct PlacementScore {
pub locality: Score,
pub cache: Score,
pub headroom: Score,
pub queue: Score,
pub reliability: Score,
pub fairness: Score,
pub cost: Score,
pub affinity: Score,
}
}27. Weighted Score
total =
W_locality * locality
+ W_cache * cache
+ W_headroom * headroom
+ W_queue * queue
+ W_reliability * reliability
+ W_fairness * fairness
+ W_cost * cost
+ W_affinity * affinity
Weights are policy/configurable.
28. Hard/Soft Separation
A high locality score can NEVER override:
missing Xcode
not enough memory
wrong OS
wrong trust class
29. Locality
Locality sources:
source snapshot present
toolchain present
dependency closure present
artifact inputs present
site-local CAS
region-local CAS
device proximity
30. Locality Hint
#![allow(unused)]
fn main() {
pub struct LocalityHints {
pub required_objects: Vec<CasObjectId>,
pub preferred_region: Option<RegionId>,
pub site: Option<SiteId>,
}
}31. CAS Locality
Scheduler may ask data plane:
which objects are already present where?
32. Locality Accuracy
Locality data can be stale.
Therefore:
locality affects score, not correctness
Runner still fetches/verifies missing objects.
33. Cache Warmth
Runner can report:
toolchain IDs
known dependency closures
action-cache hints
Use bounded summaries.
34. Avoid Huge Capability Reports
Do not report millions of CAS object IDs in heartbeat.
Use:
site cache presence
toolchain set
Bloom hints
recent closure IDs
35. Headroom Score
Prefer runner with enough remaining capacity after placement.
Avoid fragmentation.
36. Resource Fragmentation
Example:
one 64 GB runner
many 8 GB jobs
Do not consume scarce high-memory runner unnecessarily if standard runner fits.
37. Scarcity-Aware Scoring
Capabilities can have scarcity weight:
rare GPU
rare Xcode version
signing hardware
Prefer preserving rare capacity for jobs needing it.
38. Runner Reliability Score
Based on recent:
lost attempts
sandbox failures
input fetch failures
disconnects
39. Reliability Guardrail
Do not permanently starve temporarily degraded runners.
Decay historical penalties over time.
40. Retry-Aware Placement
If previous attempt failed due to runner-specific infrastructure:
prefer a different runner
41. Anti-Affinity
#![allow(unused)]
fn main() {
pub struct RetryAntiAffinity {
pub avoid_runners: BTreeSet<RunnerId>,
pub avoid_sites: BTreeSet<SiteId>,
}
}42. Same-Runner Retry
Allowed if failure class suggests workload issue or no alternative exists.
43. Failure-Domain Awareness
Possible domains:
runner
host
rack/site
region
cloud provider
44. Retry Across Failure Domains
For repeated infra failures:
move to another host/site/region
if policy/cost allows.
45. Fairness
Scheduler must prevent one tenant/project from monopolizing shared capacity.
46. Fairness Dimensions
tenant
organization
project
queue
priority class
47. Fairness Algorithm
Initial recommendation:
weighted fair queueing + aging
Simpler than overly complex dominant-resource fairness initially.
48. Weighted Fair Queueing
Each tenant/project gets configured weight.
49. Aging
Long-waiting jobs gain scheduling preference.
Prevents starvation.
50. Priority
#![allow(unused)]
fn main() {
pub enum JobPriority {
Low,
Normal,
High,
ReleaseCritical,
Emergency,
}
}51. Priority Policy
Users cannot arbitrarily elevate themselves to Emergency without permission.
52. Priority vs Fairness
Priority should influence within allowed policy but not allow permanent starvation.
53. Queue Model
Eligible jobs enter logical scheduler queues.
Potential keys:
tenant
project
capability class
priority
54. Queue Sharding
At scale, scheduler can shard by:
platform
region
capability pool
while preserving global fairness policy.
55. Initial Scheduler
Start with centralized logical scheduler service.
Avoid premature distributed peer-to-peer scheduling.
56. Scheduler Loop
load eligible batch
↓
load candidate runners
↓
hard filter
↓
score
↓
attempt lease transaction
↓
repeat
57. Batch Scheduling
Prefer small batches to reduce DB round trips.
58. Greedy Initial Placement
Initial implementation can use:
best-score greedy placement
with resource updates after each choice.
59. Global Optimization
Do not start with NP-hard global bin packing.
Add smarter batching if real measurements justify.
60. Placement Decision
#![allow(unused)]
fn main() {
pub struct PlacementDecision {
pub job: JobId,
pub runner: RunnerId,
pub score: PlacementScore,
pub reasons: Vec<PlacementReason>,
}
}61. Placement Explainability
Store/debug:
matched capabilities
top candidate scores
why winner selected
why others rejected
62. Rejection Reasons
#![allow(unused)]
fn main() {
pub enum RunnerRejectionReason {
Offline,
Draining,
MissingCapability(CapabilityId),
InsufficientCpu,
InsufficientMemory,
InsufficientDisk,
TrustMismatch,
DeviceUnavailable,
ResourceReserved,
}
}63. Scheduler Explain CLI
forgeyard scheduler explain-job <job>
Shows:
requirements
eligible runner count
rejected runners/reasons
score winner
queue age
64. No Runner Available
Job remains:
Eligible
unless queue timeout expires.
65. Unschedulable Detection
Distinguish:
temporarily no capacity
from:
no registered runner can ever satisfy requirement
66. Unschedulable State
Do not add JobState Unschedulable initially.
Keep Job Eligible + scheduler diagnostic.
67. Persistent Unschedulable Diagnostic
Store/report:
missing capability X
for UI/doctor.
68. Queue Timeout
If configured, after deadline:
Job -> Failed/TimedOut
through state service with reason:
QueueTimeout
69. Backpressure
Scheduler must slow intake when:
DB overloaded
CAS degraded
runner fleet saturated
event backlog high
70. Backpressure Classes
#![allow(unused)]
fn main() {
pub enum BackpressureSignal {
StoragePressure,
CasPressure,
RunnerSaturation,
QueueDepth,
ControlPlanePressure,
}
}71. Backpressure Action
Examples:
reduce batch size
pause low priority scheduling
delay cache warming
reject new noncritical runs at API layer
72. Scheduler Should Not Drop Jobs
Backpressure changes admission/pace, not job correctness.
73. Resource Accounting
When lease transaction commits:
runner allocated resources += job request
or equivalent reservation record.
74. Resource Reservation Record
#![allow(unused)]
fn main() {
pub struct ResourceReservation {
pub lease: LeaseId,
pub runner: RunnerId,
pub resources: ResourceVector,
}
}75. Atomicity
Lease + reservation must commit atomically.
76. Release Reservation
When lease ends:
reservation released
idempotently.
77. Lost Lease Reservation
Reconciler releases after authoritative expiry/loss.
78. Double Reservation Protection
Unique lease ID.
79. Runner Capacity Reconciliation
Compare:
persisted active reservations
vs
runner-reported active attempts
repair discrepancies.
80. Overcommit
Optional for CPU.
Default:
no memory overcommit
unless explicit policy.
81. CPU Overcommit
May be supported for lightweight jobs.
Must be bounded and observable.
82. GPU Scheduling
GPU requirements include:
vendor
model/class
VRAM
compute capability
count
exclusive/shared
83. GPU Sharing
Only if executor/platform supports safe partitioning.
84. Device Scheduling
Physical device jobs need:
runner
+
device lease
85. Device Eligibility
Examples:
Android API level
device model
ABI
iOS version
physical/simulator
86. Device Lease Coordination
Scheduler composes:
runner placement
device pool allocation
through device subsystem.
87. Two-Resource Allocation
Potential race:
runner available
device unavailable
Use reservation ordering or transactional coordinator.
88. Recommended Device Flow
select compatible runner/device pool
↓
reserve device
↓
lease job to hosting runner
If second step fails:
release device reservation
89. Signing Workers
Jobs requiring signing capability must only target restricted signing workers.
90. Trust Class
#![allow(unused)]
fn main() {
pub enum RunnerTrust {
UntrustedGeneral,
TrustedInternal,
SigningRestricted,
Confidential,
}
}91. Trust Is Hard Requirement
A normal runner cannot score its way into signing job.
92. Network Capability
Some jobs require:
internet denied
restricted egress
internal network
Scheduler must choose runner/executor supporting requested network policy.
93. Sandbox Capability
Example:
Linux namespace sandbox
Windows job object sandbox
VM isolation
confidential VM
94. Platform-Specific Toolchains
Real macOS runner requirement remains hard.
Linux cross-compilation does not satisfy macOS/Xcode production requirement where Apple tooling is needed.
95. Runner Capabilities
#![allow(unused)]
fn main() {
pub struct RunnerCapabilities {
pub platform: PlatformCapability,
pub toolchains: ToolchainCapabilitySet,
pub sandbox: SandboxCapabilitySet,
pub devices: DeviceCapabilitySummary,
pub security: SecurityCapabilitySet,
}
}96. Capability Version
Runner capability report includes schema version/digest.
97. Capability Refresh
Runner sends on:
registration
toolchain change
device change
significant resource change
not necessarily every heartbeat.
98. Capability Staleness
If report older than threshold:
runner may become unschedulable
for sensitive jobs.
99. Runner Heartbeat
Scheduler consumes liveness from runner subsystem.
100. Draining
Draining runner:
no new leases
existing jobs continue
101. Immediate Drain
Admin may request:
cancel active
through run/job service.
102. Maintenance Windows
Runner pool can be marked unavailable ahead of maintenance.
103. Runner Pools
Logical grouping:
#![allow(unused)]
fn main() {
pub struct RunnerPoolId(Ulid);
}104. Pool Uses
platform
team ownership
trust
region
cost
autoscaling
105. Pool Requirement
Job may explicitly target allowed pool set only through policy-approved config.
106. Default Pool Selection
Capability-based, not hardcoded labels.
107. Labels
Runner labels can be additional hints/constraints.
Avoid using labels to replace typed core capabilities.
108. Taints/Tolerations
Optional Kubernetes-like concept for specialized runners.
Could model as policy constraints later.
109. Initial Recommendation
Use typed capabilities + optional labels first.
110. Affinity
Soft:
same site as source
same runner as warm toolchain
111. Anti-Affinity
Soft or hard depending use:
avoid previous failed host
spread test shards
112. Test Shard Spreading
Can spread shards across failure domains to reduce correlated loss.
113. Cost-Aware Scheduling
Optional score component:
spot/cheap runner
expensive macOS
GPU cost
region egress
114. Cost Cannot Violate Requirements
Hard constraints remain first.
115. Latency-Aware Scheduling
Interactive jobs may prefer low queue/locality.
116. Batch Jobs
Nightly/background can favor cheaper capacity.
117. Scheduling Classes
#![allow(unused)]
fn main() {
pub enum SchedulingClass {
Interactive,
Normal,
Batch,
Release,
Emergency,
}
}Derived from policy/job priority.
118. Admission Control
Before becoming Eligible, project/org may enforce concurrency quotas.
Could live in scheduler/admission layer.
119. Concurrency Quotas
Examples:
max 20 running jobs/project
max 100 running jobs/tenant
max 2 release-critical jobs
120. Admission vs Placement
Admission decides:
may this job enter active scheduling?
Placement decides:
which runner?
121. Recommended Initial Design
Keep job Eligible but scheduler fairness/quota controls placement.
Add explicit admission queues later if needed.
122. Preemption
Preemption means stopping lower-priority work to free resources.
Dangerous.
123. Default
disabled
for general jobs.
124. Allowed Preemption
Potential for:
emergency
release-critical
only when policy permits.
125. Preemptibility
Job declares/plan sets:
#![allow(unused)]
fn main() {
pub enum PreemptionPolicy {
Never,
Restartable,
Checkpointable,
}
}126. Restartable
Preempted job becomes:
Lost/RetryWaiting
with infra/preemption reason.
127. Checkpointable
Future advanced capability.
Not initial requirement.
128. Preemption Cost
Score considers:
elapsed runtime
artifact progress
retry cost
priority gap
129. Never Preempt
Examples:
signing
production deploy
non-idempotent external step
130. Scheduler Persistence
Scheduler should persist only necessary coordination state:
leases
reservations
queue metadata if needed
Core queue can often be derived from Job table.
131. No Hidden In-Memory Queue Authority
In-memory heap can optimize.
Persisted Job state remains truth.
132. Scheduler Cache
In-memory:
eligible jobs
runner capabilities
resource snapshots
locality hints
rebuildable after restart.
133. Scheduler Restart
On startup:
reload active runners
reload active reservations
reload eligible jobs
reconcile
134. Distributed Scheduler
Multiple daemon replicas may run scheduler loops.
Correctness from atomic lease transaction prevents double assignment.
135. Leader vs Leaderless Scheduling
Initial distributed design can allow:
multiple scheduler workers
competing safely on store.
136. Why Not Require Raft Initially
Postgres transactional claim/lease is sufficient for job placement correctness.
Raft may later coordinate:
scheduler epoch
global exclusive operations
but not required for basic scheduling.
137. Scheduler Epoch
Optional future:
#![allow(unused)]
fn main() {
pub struct SchedulerEpoch(u64);
}Can invalidate old scheduler authority during failover.
138. Lease Authority Still Primary
Runner validates lease, not scheduler process identity.
139. Store API
Scheduler-specific atomic interface:
#![allow(unused)]
fn main() {
#[async_trait]
pub trait SchedulerStore {
async fn list_eligible_jobs(
&self,
query: EligibleJobQuery,
) -> Result<Vec<SchedulingJob>, StoreError>;
async fn try_lease_job(
&self,
command: TryLeaseJob,
) -> Result<LeaseAttemptResult, StoreError>;
}
}140. try_lease_job
Transaction validates:
job still Eligible
no termination intent
runner/session still schedulable
capacity reservation valid
then:
create attempt
create lease
reserve resources
Job -> Leased
141. Lease Race
If another scheduler leased first:
LeaseAttemptResult::Conflict
scheduler moves on.
142. Runner Capacity Race
Two schedulers targeting same runner must not over-reserve.
Use transactional resource reservation/version.
143. Runner Resource Version
#![allow(unused)]
fn main() {
pub struct RunnerResourceVersion(u64);
}or derive from active reservation rows transactionally.
144. Placement Decision Is Advisory Until Commit
The actual authority exists only after try_lease_job commits.
145. Score Recalculation
If commit conflict, recalculate next candidate.
146. Scheduler Loop Pseudocode
while capacity:
jobs = eligible_batch()
for job in fair_order(jobs):
candidates = runners()
eligible = hard_filter(job, candidates)
if eligible empty:
record diagnostic
continue
ranked = score(job, eligible)
for runner in ranked:
if try_lease(job, runner) succeeds:
dispatch lease
break
147. Dispatch Failure After Lease Commit
Lease exists but runner didn’t receive message.
Reconciliation/expiry handles.
Can retry dispatch while lease valid.
148. At-Least-Once Lease Delivery
Runner may receive duplicate lease message.
Agent deduplicates by LeaseId.
149. Lease Acceptance Timeout
If no acknowledgment quickly:
revoke/expire
according to policy.
150. Lease Revocation
Before runner starts:
Leased -> Eligible
can be safe.
After running:
use cancellation/loss semantics
151. Resource Release on Rejection
Atomic with lease rejection state transition.
152. Local Standalone Scheduler
Same algorithms with:
one local runner
Hard matching still matters.
153. Standalone Unschedulable
If project requests:
macOS
on Linux standalone:
show:
no eligible runner
rather than silently changing target.
154. Local Auto-Runner
Standalone bootstrap registers local runner capabilities automatically.
155. Distributed Agent Registration
Runner subsystem supplies capability snapshot.
156. Remote Runners
Scheduler does not assume shared filesystem.
Inputs always through CAS/source materialization.
157. Site Awareness
#![allow(unused)]
fn main() {
pub struct SiteId(Ulid);
}Examples:
home-lab
office
region-a
158. Region Awareness
#![allow(unused)]
fn main() {
pub struct RegionId(BoundedString);
}159. Network Cost
Site/region matrix can inform score.
160. Egress Avoidance
Prefer same region as large input/output durable CAS where sensible.
161. Data Size Weight
Locality benefit should consider total input bytes.
162. Input Closure Summary
Planner/data plane can provide:
required bytes
known cached bytes
163. Locality Score Example
100% local -> high score
80% local -> medium-high
0% local -> neutral
Not negative enough to violate fairness indefinitely.
164. Toolchain Warmth
Toolchain present can be significant because installation may be expensive.
165. Cache Hit Prediction
Predictive model can be used as a score hint.
Never correctness authority.
166. Scheduler Scoring Version
#![allow(unused)]
fn main() {
pub struct SchedulerPolicyVersion(Digest);
}Record for explainability.
167. Deterministic Scoring
Given same snapshot of inputs/config:
same score ordering
except explicit random tie-break if configured.
168. Tie Breaking
Recommended:
stable hash(job_id, runner_id)
for deterministic spread.
169. Avoid Runner Hotspot
Stable tie-break plus load/headroom prevents same runner always winning.
170. Fairness Queue Data
Potential:
#![allow(unused)]
fn main() {
pub struct FairnessState {
pub tenant_usage: ResourceUsage,
pub project_usage: ResourceUsage,
}
}171. CPU-Time vs Concurrent Slots
Initial fairness can use:
running job count
weighted resource units
172. Dominant Resource Fairness
Future option if heterogeneous resources create fairness issues.
173. Scheduler Quotas
#![allow(unused)]
fn main() {
pub struct SchedulerQuota {
pub max_running_jobs: Option<u32>,
pub max_cpu: Option<CpuUnits>,
pub max_memory: Option<ByteSize>,
}
}174. Quota Scope
tenant
project
runner pool
175. Quota Reconciliation
Derived from active reservations; repair if mismatch.
176. Burst
Policy may allow temporary burst above baseline.
177. Emergency Priority
Must be audited.
178. Starvation Guard
Aging ensures lower-priority jobs eventually receive opportunity unless hard capacity never exists.
179. Dedicated Runner Pools
Project may reserve runners.
Scheduler honors ownership constraints.
180. Shared Pool
Default distributed pool.
181. Reserved Capacity
Some runner capacity reserved for:
release jobs
interactive jobs
182. Capacity Class
#![allow(unused)]
fn main() {
pub enum CapacityClass {
Shared,
Reserved(ReservationGroupId),
}
}183. Spillover
Policy can allow reserved pool idle capacity to serve shared jobs.
184. Spot/Preemptible Runners
Mark runner capacity as:
preemptible infrastructure
Scheduler prefers for restartable batch jobs.
185. Spot Avoidance
Do not place signing/release-critical long job on volatile runner unless policy allows.
186. Reliability Class
#![allow(unused)]
fn main() {
pub enum RunnerReliabilityClass {
Stable,
Standard,
Preemptible,
}
}187. Scheduler Policy Input
#![allow(unused)]
fn main() {
pub struct SchedulingPolicy {
pub score_weights: ScoreWeights,
pub quotas: QuotaPolicy,
pub preemption: PreemptionConfig,
pub locality: LocalityPolicy,
}
}188. Config vs Policy
Admin config sets operational defaults.
Policy engine can impose security/business constraints.
189. Policy Example
production deploy:
only trusted pool
preemption forbidden
region = primary
190. Scheduler Does Not Evaluate User Authorization
Authz already happened upstream.
It applies scheduling policy to already-authorized job.
191. Capability Requirement Provenance
For explainability, each requirement records source:
#![allow(unused)]
fn main() {
pub enum CapabilityRequirementOrigin {
Pipeline,
Ecosystem,
Platform,
Policy,
Device,
Security,
}
}192. Explain Missing Capability
Example:
requires Xcode 18
origin: Swift iOS build plan
193. Scheduler Diagnostics
#![allow(unused)]
fn main() {
pub struct SchedulingDiagnostic {
pub job: JobId,
pub code: SchedulingDiagnosticCode,
pub details: Vec<SchedulingDetail>,
}
}194. Diagnostic Codes
Examples:
NO_RUNNERS
NO_MATCHING_PLATFORM
INSUFFICIENT_MEMORY
MISSING_TOOLCHAIN
NO_DEVICE
TRUST_MISMATCH
QUOTA_BLOCKED
195. Diagnostic Persistence
Store latest summary, not every scheduling loop iteration.
196. Diagnostic Debounce
Avoid event spam for persistent unschedulable job.
197. Scheduler Doctor
forgeyard scheduler doctor
Checks:
runner pools
missing capability classes
quota config
stuck eligible jobs
lease backlog
resource accounting
198. CLI
forgeyard scheduler status
forgeyard scheduler explain-job
forgeyard scheduler queues
forgeyard scheduler runners
forgeyard scheduler capacity
forgeyard scheduler rebalance
199. scheduler status
Shows:
eligible jobs
running jobs
queue age
active leases
runner capacity
backpressure state
200. scheduler queues
Grouped by:
priority
tenant
platform
pool
201. scheduler capacity
Shows:
CPU
memory
disk
GPU
device
platform pools
202. rebalance
Should not move running jobs by default.
May only reconsider queued/eligible jobs.
203. Dioxus UI
Scheduler page:
Overview
Queues
Runner Pools
Capacity
Unschedulable
Leases
Fairness
204. Queue Visualization
Show:
waiting count
oldest wait
priority mix
205. Runner Pool View
Show:
online/draining/offline
capacity
allocatable
active jobs
toolchains
trust
region/site
206. Unschedulable View
Human explanation:
7 jobs require macOS/Xcode 18
0 compatible runners online
207. API
Potential admin/read APIs:
GET /v1/scheduler/status
GET /v1/scheduler/queues
GET /v1/scheduler/capacity
GET /v1/jobs/{id}/scheduling
208. Internal Scheduler Control
Not broadly public:
drain runner
set pool state
through runner/admin APIs.
209. Metrics
scheduler_cycle_latency
scheduler_jobs_considered
scheduler_placements
scheduler_placement_conflicts
scheduler_unschedulable
scheduler_queue_wait
scheduler_fairness_delay
scheduler_backpressure
scheduler_preemptions
scheduler_lease_dispatch_failures
210. Resource Metrics
runner_cpu_allocated
runner_memory_allocated
runner_disk_allocated
gpu_allocated
device_allocated
211. Locality Metrics
placement_locality_score
cas_local_hit_after_placement
toolchain_warm_hit
212. Reliability Metrics
runner_lost_attempt_rate
runner_rejection_rate
213. Tracing
Spans:
scheduler.cycle
scheduler.eligibility
scheduler.score
scheduler.place
scheduler.lease
scheduler.dispatch
scheduler.reconcile
214. High-Cardinality Caution
Do not put RunnerId/JobId into metrics labels.
Use tracing/logs for IDs.
215. Alerting
Alert on:
eligible queue growing
oldest wait above SLO
lease conflict spike
resource accounting mismatch
no runners for critical capability
scheduler loop stalled
216. Scheduler Reconciler
forgeyard-scheduler-reconcile
Checks:
active reservations without active lease
lease without reservation
runner over-allocation
eligible job never reconsidered
draining runner receiving new lease
217. Reservation Repair
Recompute from authoritative active leases if needed.
218. Queue Recovery
Eligible jobs are reloaded after restart.
No lost in-memory queue.
219. Scheduler Crash During Placement
If lease transaction not committed:
nothing authoritative happened
If committed but dispatch not sent:
re-dispatch/reconcile
220. DB Failure
Scheduler pauses new placements.
Existing jobs continue under valid leases.
221. CAS Degradation
If durable CAS unavailable for required job inputs:
scheduler may pause placement depending data-plane health/policy.
222. Runner Registration Race
New runner becomes eligible after registration transaction + health state.
223. Runner Disconnect
No new jobs.
Existing leases handled by run/lease expiration logic.
224. Scheduler and Run State Boundary
Scheduler may request:
TryLeaseJob
It may not directly set:
JobState::Leased
outside state/store API.
225. Scheduler and Runner Boundary
Scheduler dispatches lease payload.
Runner acceptance/rejection goes through run/job state service.
226. Scheduler and Device Boundary
Device subsystem provides reservable device capability.
Scheduler orchestrates combined placement.
227. Scheduler and Autoscaling
Future autoscaler observes:
queue demand
capability shortages
and scales runner pools.
228. Autoscaler Is Separate
Suggested future crates:
crates/autoscale/
├── forgeyard-autoscale/
├── forgeyard-autoscale-demand/
├── forgeyard-autoscale-kubernetes/
└── forgeyard-autoscale-cloud/
229. Scheduler Demand Signal
#![allow(unused)]
fn main() {
pub struct CapacityDemand {
pub capability_class: CapabilityClass,
pub queued_jobs: u32,
pub requested_resources: ResourceVector,
}
}230. KEDA/Kubernetes
Optional adapter consumes demand.
Scheduler itself remains infrastructure-neutral.
231. Scale-to-Zero
For expensive pools, autoscaler may scale to zero.
Scheduler reports jobs waiting for that capability.
232. Startup Latency
Queue timeout must account for autoscaling cold-start policy if configured.
233. Scheduler API Trait
#![allow(unused)]
fn main() {
#[async_trait]
pub trait Scheduler {
async fn schedule_once(
&self,
context: SchedulingContext,
) -> Result<SchedulingCycleResult, SchedulerError>;
}
}234. Eligibility Trait
#![allow(unused)]
fn main() {
pub trait EligibilityEngine {
fn evaluate(
&self,
job: &SchedulingJob,
runner: &SchedulingRunner,
) -> EligibilityResult;
}
}235. Scoring Trait
#![allow(unused)]
fn main() {
pub trait PlacementScorer {
fn score(
&self,
job: &SchedulingJob,
runner: &SchedulingRunner,
) -> PlacementScore;
}
}236. Pluggable Scoring
Internal strategy can evolve behind stable trait.
Do not expose arbitrary third-party scheduler code initially.
237. Deterministic Test Scorer
Testkit can use fixed scorer.
238. Scheduler Error Model
#![allow(unused)]
fn main() {
pub enum SchedulerError {
StoreUnavailable,
InvalidJobRequirements,
ResourceAccountingCorrupt,
PlacementConflict,
PolicyError,
Internal,
}
}239. Placement Conflict Is Normal
Not exceptional/fatal in multi-scheduler environment.
240. Retry Classification
store unavailable -> backoff
placement conflict -> immediate next candidate
invalid requirements -> no retry until plan fixed
241. Testkit
forgeyard-scheduler-testkit/src/
├── lib.rs
├── job.rs
├── runner.rs
├── capabilities.rs
├── resources.rs
├── scorer.rs
├── fairness.rs
└── assertions.rs
242. Unit Tests
Test:
hard capability matching
resource fit
score ordering
priority
fairness
aging
retry anti-affinity
drain behavior
243. Property Tests
Properties:
ineligible runner never selected
selected runner always satisfies hard requirements
resource reservation never exceeds capacity
aging prevents starvation under bounded assumptions
244. Concurrency Tests
Simulate:
two schedulers
same job
same runner
many jobs
prove one lease per job and no over-reservation.
245. Failure Injection
Inject:
DB timeout
runner disconnect
dispatch failure
stale capability report
CAS locality mismatch
246. Large Fleet Simulation
Simulate:
10k runners
100k eligible jobs
for algorithmic profiling.
247. Scheduler Complexity
Initial hard filtering should avoid:
jobs × every runner
at very large scale.
248. Capability Index
Maintain runner indexes by:
OS
architecture
pool
trust
toolchain class
device class
249. Candidate Narrowing
Use indexes to produce candidate subset before detailed matching.
250. Exact Match Still Required
Index is optimization.
Matcher verifies.
251. Resource Index
Keep available-capacity buckets or sorted structures in memory.
Rebuildable.
252. Scheduler Snapshot
Each cycle uses coherent-enough snapshot of:
runner state
reservations
eligible jobs
Final lease transaction resolves races.
253. Stale In-Memory View
Safe because commit revalidates.
254. Queue Ordering
Suggested base key:
effective_priority
aging
created_at
job_id
255. Effective Priority
base priority
+
aging
+
policy adjustments
bounded.
256. Starvation Bound
Document/test expected fairness characteristics.
257. Large Job Avoidance
Small jobs should not indefinitely starve large resource jobs.
Reserve/aging can help.
258. Gang Scheduling
Jobs needing multiple runners simultaneously are not initially required.
259. Future Distributed Tests
If MPI/multi-node workloads added, separate gang scheduling subsystem.
260. Service Affinity
If one job depends on another runtime service on same host, explicit co-location requirement.
Avoid implicit shared-host assumptions.
261. Build Shards
Independent shards scheduled independently.
262. Matrix Jobs
Ordinary jobs after pipeline expansion.
Scheduler sees only capabilities/resources.
263. Reproducibility Jobs
May require:
different runner/site
to strengthen independent reproduction.
This is a hard/soft anti-affinity from reproducibility policy.
264. Multi-Party Reproduction
Require distinct trust/failure domains if configured.
265. Security-Sensitive Jobs
Examples:
signing
secret-heavy integration
production deployment
must run only in allowed trust pools.
266. Untrusted Proposal Jobs
Should prefer/general-purpose untrusted pools and never privileged workers.
267. Runner Pool Isolation
Pool policy rejects incompatible trust class.
268. Resource Estimate Feedback
Actual runtime metrics can improve future requested recommendations.
But scheduler never silently lowers declared limits below policy.
269. Auto-Tuning
Optional:
suggest memory request
suggest CPU
not mutate build semantics automatically at first.
270. Queue Prediction
Estimate wait time.
Informational only.
271. ETA
Use historical stats but label as estimate.
272. Scheduler History
Store summarized placement history if useful for analytics.
Do not store every score for every candidate indefinitely.
273. Placement Audit
For security-critical jobs, record:
selected runner
trust
policy
capability digest
274. Release Audit
Release/signing placement should be auditable.
275. Runner Capability Digest
Attempt record can reference exact capability snapshot digest used for placement.
276. Runner Software Version
Hard compatibility requirement between daemon/agent.
Transport/version layer handles.
277. Mixed Agent Versions
Scheduler excludes incompatible agents.
278. Runner Health
Runner subsystem reports:
Healthy
Degraded
Unhealthy
279. Degraded Runner
May receive only low-risk jobs or none depending policy.
280. Temperature/Hardware Health
Optional specialized capability/health inputs.
281. Disk Pressure
Runner enters:
Saturated
or reduces allocatable disk.
282. Memory Pressure
Same.
283. Local CAS Pressure
Can reduce locality benefit or block large jobs if no staging capacity.
284. Scheduler Backoff
If no placement possible:
do not spin aggressively.
Use event-driven wakeups + bounded polling/reconcile fallback.
285. Wakeup Sources
job becomes Eligible
runner registers
runner frees resources
retry deadline expires
device becomes available
286. Event-Driven Scheduling
Preferred for responsiveness.
287. Polling Fallback
Periodic scan ensures missed events do not stall jobs.
288. Scheduling Trigger Coalescing
Many events can collapse into one cycle.
289. Scheduler Work Queue
Internal in-memory wake queue.
Not durable authority.
290. Lease Dispatch Queue
Could use outbox/event mechanism.
291. Dispatch Outbox
Transaction:
lease commit
+
dispatch event
then transport sends.
Improves reliability.
292. Duplicate Dispatch
Runner deduplicates by LeaseId.
293. Lease Acknowledgement
Runner sends:
accept/reject
294. No Ack
Lease expiry/revoke.
295. Lease TTL
Depends on:
dispatch latency
network conditions
job type
Separate acceptance TTL and execution-renewal TTL can be useful.
296. Lease Phases
Potential:
OfferLease
ActiveLease
Initial implementation can use one lease + ack deadline.
297. Offer Timeout
If not accepted quickly:
return Job Eligible
298. Active Renewal
After accept, periodic heartbeat renews.
299. Scheduler Does Not Own Lease Renewal
Lease/run service owns authoritative renewal.
Scheduler may observe.
300. Scheduler Rebalance
Do not revoke healthy running leases just because a better runner appears.
301. Queue Reordering
Eligible jobs can be reprioritized by policy/admin.
302. Priority Change Audit
Required.
303. Scheduler Security Invariants
- only authenticated registered runners considered;
- trust requirement is hard;
- device/signing capabilities cannot be spoofed without registration validation;
- scheduler never bypasses policy to improve throughput;
- tenant quota enforced;
- runner labels cannot grant privileged trust.
304. Capability Attestation
High-trust runners may require stronger enrollment/cert identity.
Scheduler consumes trust result, not attestation implementation.
305. Agent Self-Reported Capability Trust
General capabilities can be self-reported + probed.
Sensitive capabilities should be provisioned/verified.
306. Toolchain Verification
Runner reports immutable toolchain identity.
If mismatch at runtime, attempt fails preparation.
307. Scheduler Correctness Invariants
- only
Eligiblejobs are placed; - selected runner satisfies all hard requirements;
- resource reservation never exceeds allowed capacity;
- one authoritative lease per current attempt;
- placement is not authoritative until transaction commit;
- soft score never overrides hard rejection;
- locality is advisory;
- predictive models are advisory;
- in-memory state is rebuildable;
- multi-daemon races resolve through store.
308. Standalone Invariants
- same eligibility/score logic;
- one local runner can still be rejected;
- no silent target substitution;
- local mode may optimize transport but not semantics.
309. Distributed Invariants
- agents never write scheduler DB directly;
- scheduler can restart without losing jobs;
- lease dispatch is at-least-once safe;
- runner churn handled through liveness/lease expiry;
- no Raft required for basic placement correctness.
310. Implementation Phase 1 — Models
Implement:
SchedulingJob
SchedulingRunner
ResourceVector
PlacementScore
RunnerRejectionReason
311. Phase 2 — Hard Matcher
Implement:
platform
architecture
resources
toolchain
trust
sandbox
312. Phase 3 — Simple Scorer
Implement:
headroom
load
locality
stable tie-break
313. Phase 4 — Lease Integration
Implement try_lease_job with atomic attempt/resource reservation.
314. Phase 5 — Runner Pools / Drain
Implement pool grouping and drain states.
315. Phase 6 — Fairness / Priority
Add weighted queues + aging.
316. Phase 7 — Retry-Aware Placement
Add anti-affinity/failure-domain behavior.
317. Phase 8 — Device/GPU
Add specialized resource scheduling.
318. Phase 9 — Backpressure
Integrate control-plane/CAS/runner pressure signals.
319. Phase 10 — Reconciliation / HA
Multi-scheduler contention tests, rebuildable caches, reservation repair.
320. Phase 11 — Advanced Optimization
Add:
scarcity
cost
predictive cache
autoscaling demand
only after correctness stable.
321. Acceptance Tests
- Ineligible runner is never selected.
- Missing toolchain rejects runner.
- Wrong OS rejects runner.
- Insufficient memory rejects runner.
- Trust mismatch rejects runner.
- Eligible runner can be scored.
- Highest score wins absent fairness constraints.
- Hard requirement always dominates soft score.
- Resource reservation is atomic with lease.
- Two schedulers cannot over-reserve same runner.
- Two schedulers cannot lease same job twice.
- Draining runner receives no new job.
- Offline runner receives no new job.
- Retry after runner loss prefers different runner when possible.
- Locality improves score but missing local object never breaks execution.
- Stale locality hint only causes slower fetch, not incorrect placement.
- Fairness prevents one tenant from monopolizing bounded shared capacity.
- Aging prevents indefinite starvation.
- Priority elevation requires policy/permission.
- Queue timeout is surfaced explicitly.
- Unschedulable job remains Eligible with diagnostic.
- Standalone Linux refuses macOS-only job.
- Real Apple capability required for Apple-specific job.
- Device job acquires compatible device+runner.
- Signing job cannot run on general runner.
- Scheduler restart rebuilds queue from persisted state.
- Lease committed but dispatch lost is safely retried.
- Duplicate lease dispatch is safe.
- DB outage pauses new placements without corrupting active leases.
- Reconciler repairs leaked resource reservation.
- Predictive cache score cannot bypass hard requirements.
- Same scheduler logic works with Stoolap and Postgres.
- Scheduler explain output identifies rejection reasons.
- High queue load does not require full runner scan per job after indexing.
- Forgeyard self-hosting jobs can be placed through this scheduler.
322. Production Readiness Gates
Do not call scheduler production-ready until:
hard matcher correct
resource reservation atomic
multi-scheduler race tests pass
runner drain works
fairness/priority behavior tested
lease dispatch recovery works
retry-aware placement works
reconciliation works
unschedulable diagnostics exist
metrics/doctor exist
Advanced preemption/autoscaling/predictive scoring may mature later.
323. Architectural Invariants
- Scheduler consumes only Eligible jobs.
- Scheduler never directly assigns Job state.
- Attempt+Lease creation is authoritative store operation.
- Hard filtering precedes scoring.
- Every required capability must match.
- Soft scoring never relaxes correctness.
- Resource capacity is reserved transactionally.
- Placement decision is advisory until commit.
- In-memory scheduler state is rebuildable.
- Locality/cache data is advisory.
- Predictive models are advisory.
- Runner trust is hard requirement.
- Draining runners receive no new work.
- Retry placement may avoid prior failure domains.
- Fairness and priority are explicit policy.
- Preemption is disabled by default.
- Signing/device/confidential jobs use restricted capability paths.
- Standalone and distributed modes share scheduler semantics.
- Multiple schedulers may contend safely.
- Scheduler does not require Raft for basic correctness.
- Agent never directly edits scheduler metadata.
- Resource leaks are reconciled.
- Dispatch is at-least-once safe.
- Queue persistence derives from Job state, not hidden heap authority.
- Forgeyard itself should dogfood the scheduler.
324. Final Target Architecture
Eligible Job Set
│
▼
Fair Queue Order
│
▼
Candidate Runner Index
│
▼
Hard Eligibility
┌─────────┼─────────┐
▼ ▼ ▼
Platform Resources Trust
│ │ │
└─────────┼─────────┘
▼
Eligible Set
│
▼
Soft Score
┌──────────────┼──────────────┐
▼ ▼ ▼
Locality Headroom Fairness
│ │ │
└──────────────┼──────────────┘
▼
Best Placement
│
▼
try_lease_job()
│
┌────────────────┴────────────────┐
▼ ▼
Conflict Commit
│ │
└── retry candidate ▼
Attempt + Lease
│
▼
Runner
325. Final Architectural Position
Scheduling correctness:
Eligible Job
+
Hard Requirements
+
Current Runner Capabilities
+
Current Allocatable Resources
↓
Eligible Runner Set
Optimization:
Eligible Runner Set
+
Locality
+
Cache warmth
+
Headroom
+
Reliability
+
Fairness
+
Cost
↓
Ranked Candidates
Authority:
Ranked Candidate
↓
atomic attempt + lease + reservation transaction
↓
only then placement becomes real
The key guarantee is:
Forgeyard can optimize aggressively for locality, speed, fairness, cache warmth, and cost without ever letting those optimizations weaken the hard correctness, trust, platform, or resource requirements of the job.
326. New-Repository Sequence
The sequence is now:
01 Core Domain & Foundation
02 Storage & Metadata
03 CAS & Artifact Data Plane
04 Pipeline IR / Parsing / Planning
05 Run / Job State Machine
06 Scheduler
07 Runner / Agent
08 Sandbox & Executor
09 Transport / QUIC / Protocol
10 Events / Reconciliation
11 Policy / Authorization / Identity
12 Secrets & Trust
13 Supply Chain / SBOM / Provenance / Signing
14 Packaging
15 Release
16 Deployment
17 Observability / Health / Doctor
18 API / Axum
19 Dioxus UI
20 Device Lab
21 SCM Provider Integrations
22 HA / Coordination / Raft
23 RBE Interop
24 Plugin / Extension Architecture
25 Operations / Backup / Upgrade / DR
26 Self-Hosting / Bootstrap / Release of Forgeyard