cooperative-rate-limiting.md (7451B)
1 # Cooperative Rate Limiting for Multi-Agent Deployments 2 3 > Coordinate API quota across multiple Ralph instances to prevent cascading failures. 4 5 ## Problem 6 7 The [circuit breaker template](ralph-circuit-breaker.md) handles single-instance rate limiting well. But when multiple Ralphs run across machines (or pods on K8s), each instance independently hits API limits: 8 9 - **No coordination** — 5 Ralphs each think they have full API quota 10 - **Thundering herd** — All Ralphs retry simultaneously after rate limit resets 11 - **Priority inversion** — Low-priority work exhausts quota before critical work runs 12 - **Reactive only** — Circuit opens AFTER 429, wasting the failed request 13 14 ## Solution: 6-Pattern Architecture 15 16 These patterns layer on top of the existing circuit breaker. Each is independent — adopt one or all. 17 18 ### Pattern 1: Traffic Light (RAAS — Rate-Aware Agent Scheduling) 19 20 Map GitHub API `X-RateLimit-Remaining` to traffic light states: 21 22 | State | Remaining % | Behavior | 23 |-------|------------|----------| 24 | 🟢 GREEN | >20% | Normal operation | 25 | 🟡 AMBER | 5–20% | Only P0 agents proceed | 26 | 🔴 RED | <5% | Block all except emergency P0 | 27 28 ```typescript 29 type TrafficLight = 'green' | 'amber' | 'red'; 30 31 function getTrafficLight(remaining: number, limit: number): TrafficLight { 32 const pct = remaining / limit; 33 if (pct > 0.20) return 'green'; 34 if (pct > 0.05) return 'amber'; 35 return 'red'; 36 } 37 38 function shouldProceed(light: TrafficLight, agentPriority: number): boolean { 39 if (light === 'green') return true; 40 if (light === 'amber') return agentPriority === 0; // P0 only 41 return false; // RED — block all 42 } 43 ``` 44 45 ### Pattern 2: Cooperative Token Pool (CMARP) 46 47 A shared JSON file (`~/.squad/rate-pool.json`) distributes API quota: 48 49 ```json 50 { 51 "totalLimit": 5000, 52 "resetAt": "2026-03-22T20:00:00Z", 53 "allocations": { 54 "picard": { "priority": 0, "allocated": 2000, "used": 450, "leaseExpiry": "2026-03-22T19:55:00Z" }, 55 "data": { "priority": 1, "allocated": 1750, "used": 200, "leaseExpiry": "2026-03-22T19:55:00Z" }, 56 "ralph": { "priority": 2, "allocated": 1250, "used": 100, "leaseExpiry": "2026-03-22T19:55:00Z" } 57 } 58 } 59 ``` 60 61 **Rules:** 62 - P0 agents (Lead) get 40% of quota 63 - P1 agents (specialists) get 35% 64 - P2 agents (Ralph, Scribe) get 25% 65 - Stale leases (>5 minutes without heartbeat) are auto-recovered 66 - Each agent checks their remaining allocation before making API calls 67 68 ```typescript 69 interface RatePoolAllocation { 70 priority: number; 71 allocated: number; 72 used: number; 73 leaseExpiry: string; 74 } 75 76 interface RatePool { 77 totalLimit: number; 78 resetAt: string; 79 allocations: Record<string, RatePoolAllocation>; 80 } 81 82 function canUseQuota(pool: RatePool, agentName: string): boolean { 83 const alloc = pool.allocations[agentName]; 84 if (!alloc) return true; // Unknown agent — allow (graceful) 85 86 // Reclaim stale leases from crashed agents 87 const now = new Date(); 88 for (const [name, a] of Object.entries(pool.allocations)) { 89 if (new Date(a.leaseExpiry) < now && name !== agentName) { 90 a.allocated = 0; // Reclaim 91 } 92 } 93 94 return alloc.used < alloc.allocated; 95 } 96 ``` 97 98 ### Pattern 3: Predictive Circuit Breaker (PCB) 99 100 Opens the circuit BEFORE getting a 429 by predicting when quota will run out: 101 102 ```typescript 103 interface RateSample { 104 timestamp: number; // Date.now() 105 remaining: number; // from X-RateLimit-Remaining header 106 } 107 108 class PredictiveCircuitBreaker { 109 private samples: RateSample[] = []; 110 private readonly maxSamples = 10; 111 private readonly warningThresholdSeconds = 120; 112 113 addSample(remaining: number): void { 114 this.samples.push({ timestamp: Date.now(), remaining }); 115 if (this.samples.length > this.maxSamples) { 116 this.samples.shift(); 117 } 118 } 119 120 /** Predict seconds until quota exhaustion using linear regression */ 121 predictExhaustion(): number | null { 122 if (this.samples.length < 3) return null; 123 124 const n = this.samples.length; 125 const first = this.samples[0]; 126 const last = this.samples[n - 1]; 127 128 const elapsedMs = last.timestamp - first.timestamp; 129 if (elapsedMs === 0) return null; 130 131 const consumedPerMs = (first.remaining - last.remaining) / elapsedMs; 132 if (consumedPerMs <= 0) return null; // Not consuming — safe 133 134 const msUntilExhausted = last.remaining / consumedPerMs; 135 return msUntilExhausted / 1000; 136 } 137 138 shouldOpen(): boolean { 139 const eta = this.predictExhaustion(); 140 if (eta === null) return false; 141 return eta < this.warningThresholdSeconds; 142 } 143 } 144 ``` 145 146 ### Pattern 4: Priority Retry Windows (PWJG) 147 148 Non-overlapping jitter windows prevent thundering herd: 149 150 | Priority | Retry Window | Description | 151 |----------|-------------|-------------| 152 | P0 (Lead) | 500ms–5s | Recovers first | 153 | P1 (Specialists) | 2s–30s | Moderate delay | 154 | P2 (Ralph/Scribe) | 5s–60s | Most patient | 155 156 ```typescript 157 function getRetryDelay(priority: number, attempt: number): number { 158 const windows: Record<number, [number, number]> = { 159 0: [500, 5000], // P0: 500ms–5s 160 1: [2000, 30000], // P1: 2s–30s 161 2: [5000, 60000], // P2: 5s–60s 162 }; 163 164 const [min, max] = windows[priority] ?? windows[2]; 165 const base = Math.min(min * Math.pow(2, attempt), max); 166 const jitter = Math.random() * base * 0.5; 167 return base + jitter; 168 } 169 ``` 170 171 ### Pattern 5: Resource Epoch Tracker (RET) 172 173 Heartbeat-based lease system for multi-machine deployments: 174 175 ```typescript 176 interface ResourceLease { 177 agent: string; 178 machine: string; 179 leaseStart: string; 180 leaseExpiry: string; // Typically 5 minutes from now 181 allocated: number; 182 } 183 184 // Each agent renews its lease every 2 minutes 185 // If lease expires (agent crashed), allocation is reclaimed 186 ``` 187 188 ### Pattern 6: Cascade Dependency Detector (CDD) 189 190 Track downstream failures and apply backpressure: 191 192 ``` 193 Agent A (rate limited) → Agent B (waiting for A) → Agent C (waiting for B) 194 ↑ Backpressure signal: "don't start new work" 195 ``` 196 197 When a dependency is rate-limited, upstream agents should pause new work rather than queuing requests that will fail. 198 199 ## Kubernetes Integration 200 201 On K8s, cooperative rate limiting can use KEDA to scale pods based on API quota: 202 203 ```yaml 204 apiVersion: keda.sh/v1alpha1 205 kind: ScaledObject 206 spec: 207 scaleTargetRef: 208 name: ralph-deployment 209 triggers: 210 - type: external 211 metadata: 212 scalerAddress: keda-copilot-scaler:6000 213 # Scaler returns 0 when rate limited → pods scale to zero 214 ``` 215 216 See [keda-copilot-scaler](https://github.com/tamirdresher/keda-copilot-scaler) for a complete implementation. 217 218 ## Quick Start 219 220 1. **Minimum viable:** Adopt Pattern 1 (Traffic Light) — read `X-RateLimit-Remaining` from API responses 221 2. **Multi-machine:** Add Pattern 2 (Cooperative Pool) — shared `rate-pool.json` 222 3. **Production:** Add Pattern 3 (Predictive CB) — prevent 429s entirely 223 4. **Kubernetes:** Add KEDA scaler for automatic pod scaling 224 225 ## References 226 227 - [Circuit Breaker Template](ralph-circuit-breaker.md) — Foundation patterns 228 - [Squad on AKS](https://github.com/tamirdresher/squad-on-aks) — Production K8s deployment 229 - [KEDA Copilot Scaler](https://github.com/tamirdresher/keda-copilot-scaler) — Custom KEDA external scaler