Dig notes from the Sandstone layer
Pulse
A load balancer I wrote in Go, in front of three API replicas that share a Redis rate limiter and cache. I built it to find out where a small distributed system breaks first. It was the load balancer.
- Stack
- Go, Redis (Lua), Prometheus, Docker Compose, k6
- When
- October 2026
- Measured on
- MacBook Air, Docker Desktop capped at 8 CPUs and 4 GB, k6 sharing those CPUs
Why I built it
I built Pulse while applying for backend and infrastructure internships. I didn’t want a service that only works on a quiet laptop. I wanted to know where it fails, so the load tests are there to find the breaking point, not to produce a flattering number.
How it fits together
k6 sends GET /items/{id} at a fixed arrival rate to the load balancer on port 8000. The balancer round-robins with an atomic counter, health-checks each backend every 2 seconds, and also marks a backend down when a proxied request errors.
Three stateless API replicas handle the requests. All shared state lives in Redis: the rate-limit counters and a 30-second item cache, in front of a simulated 5 ms database. Any replica can serve any request. Prometheus scrapes each replica every 5 seconds.
Decisions, and what each one cost
| Decision | Why | Cost |
|---|---|---|
| Rate limit in one Redis Lua script (INCR + PEXPIRE), 1,000 req/s per client | Atomic: two replicas can’t both allow request 1,000, and a crash can’t leave a counter with no expiry | A fixed window can let through up to twice the limit across a window boundary |
| Limiter fails closed | If Redis is down the API returns 503, which protects whatever is downstream | A Redis outage becomes a full /items outage, even though the cache could have fallen back |
| Cache-aside, 30 s TTL | On a Redis error it falls back to the store, so the cache is never a hard dependency | Data can be up to 30 seconds stale |
| Client identity from an X-Client-ID header | The balancer hides client IPs | Anyone can spoof it; production needs an API key or authenticated identity |
| Round-robin on an atomic counter | No data race and no lock on the hot path | Ignores how loaded each backend is |
| Open-model load test (fixed arrival rate) | Shows queueing that closed-model tests hide | Needs enough pre-allocated virtual users; I under-allocated at 2,500 req/s |
Results
Each run: 15 seconds of warm-up, then 60 measured seconds at a constant rate. A run passes if p99 is under 100 ms, errors are under 1%, and nothing is dropped.
| Target req/s | Served | p99 | Dropped | LB CPU | Each API | Redis | |
|---|---|---|---|---|---|---|---|
| 1,500 | 1,500 | 8.0 ms | 0 | 39% | ~14% | 9% | Pass |
| 2,000 | 1,904 | 333 ms | 7,105 | 96% | ~45% | 37% | Fail |
| 2,500 | 2,478 | 41 ms | 961 | 87% | ~34% | 23% | Fail |
| 3,000 | 2,916 | 156 ms | 5,953 | 89% | ~38% | 34% | Fail |
What broke
- The load balancer was the bottleneck. Under overload it sat at about one core while each API used less than half of one.
- More replicas added almost nothing. One replica served about 1,975 req/s; three served about 2,020, because every request still passes through the one balancer.
- My health checks caused an outage. Under overload /healthz took longer than the 1-second check timeout, so the balancer ejected a healthy backend and returned 503s until the next check, about 2 seconds later. A small cascading failure.
- Graceful shutdown helps but doesn’t finish the job. Stopping a replica under load gave about 0.1% 502s with SIGTERM and about 0.3% with kill -9, because requests that arrive before the next health check are still refused.
- One run proves little. The same code gave 2,423 and 1,599 req/s on two runs, so I stopped reading anything into small differences.
What I’d do next
- Eject a backend only after several consecutive failed checks, and never eject all of them.
- Drain a replica before shutting it down; retry idempotent requests.
- Profile the balancer with pprof, then run more than one.
- Swap the fixed window for a sliding one and pipeline the Redis calls.
- Add OpenTelemetry tracing.