Dig notes from the Sandstone layer

Pulse

A load balancer I wrote in Go, in front of three API replicas that share a Redis rate limiter and cache. I built it to find out where a small distributed system breaks first. It was the load balancer.

Stack
Go, Redis (Lua), Prometheus, Docker Compose, k6
When
October 2026
Measured on
MacBook Air, Docker Desktop capped at 8 CPUs and 4 GB, k6 sharing those CPUs

Code on GitHub

Pulse architecturek6 sends requests to the load balancer, which round-robins across three API replicas. All three use Redis for rate-limit counters and cache. Prometheus scrapes the replicas. CPU figures are from the 2,000 requests per second run. k6load balancerapi-1api-2api-3RedisProm. fixed rateround-robin96% CPU~45%~45%~45%37% health checks every 2 sscrape every 5 s

Why I built it

I built Pulse while applying for backend and infrastructure internships. I didn’t want a service that only works on a quiet laptop. I wanted to know where it fails, so the load tests are there to find the breaking point, not to produce a flattering number.

How it fits together

k6 sends GET /items/{id} at a fixed arrival rate to the load balancer on port 8000. The balancer round-robins with an atomic counter, health-checks each backend every 2 seconds, and also marks a backend down when a proxied request errors.

Three stateless API replicas handle the requests. All shared state lives in Redis: the rate-limit counters and a 30-second item cache, in front of a simulated 5 ms database. Any replica can serve any request. Prometheus scrapes each replica every 5 seconds.

Decisions, and what each one cost

DecisionWhyCost
Rate limit in one Redis Lua script (INCR + PEXPIRE), 1,000 req/s per clientAtomic: two replicas can’t both allow request 1,000, and a crash can’t leave a counter with no expiryA fixed window can let through up to twice the limit across a window boundary
Limiter fails closedIf Redis is down the API returns 503, which protects whatever is downstreamA Redis outage becomes a full /items outage, even though the cache could have fallen back
Cache-aside, 30 s TTLOn a Redis error it falls back to the store, so the cache is never a hard dependencyData can be up to 30 seconds stale
Client identity from an X-Client-ID headerThe balancer hides client IPsAnyone can spoof it; production needs an API key or authenticated identity
Round-robin on an atomic counterNo data race and no lock on the hot pathIgnores how loaded each backend is
Open-model load test (fixed arrival rate)Shows queueing that closed-model tests hideNeeds enough pre-allocated virtual users; I under-allocated at 2,500 req/s

Results

Each run: 15 seconds of warm-up, then 60 measured seconds at a constant rate. A run passes if p99 is under 100 ms, errors are under 1%, and nothing is dropped.

Target req/sServedp99DroppedLB CPUEach APIRedis
1,5001,5008.0 ms039%~14%9%Pass
2,0001,904333 ms7,10596%~45%37%Fail
2,5002,47841 ms96187%~34%23%Fail
3,0002,916156 ms5,95389%~38%34%Fail

What broke

  • The load balancer was the bottleneck. Under overload it sat at about one core while each API used less than half of one.
  • More replicas added almost nothing. One replica served about 1,975 req/s; three served about 2,020, because every request still passes through the one balancer.
  • My health checks caused an outage. Under overload /healthz took longer than the 1-second check timeout, so the balancer ejected a healthy backend and returned 503s until the next check, about 2 seconds later. A small cascading failure.
  • Graceful shutdown helps but doesn’t finish the job. Stopping a replica under load gave about 0.1% 502s with SIGTERM and about 0.3% with kill -9, because requests that arrive before the next health check are still refused.
  • One run proves little. The same code gave 2,423 and 1,599 req/s on two runs, so I stopped reading anything into small differences.

What I’d do next

  • Eject a backend only after several consecutive failed checks, and never eject all of them.
  • Drain a replica before shutting it down; retry idempotent requests.
  • Profile the balancer with pprof, then run more than one.
  • Swap the fixed window for a sliding one and pipeline the Redis calls.
  • Add OpenTelemetry tracing.