How We Reduced API Latency by 40%

The Challenge

As our user base grew, we noticed increasing tail latency across our API gateway. While median response times remained healthy, p99 latency had crept up to 85ms — unacceptable for real-time applications.

We set a target: reduce p99 latency to under 50ms without compromising reliability or increasing infrastructure costs.

Root Cause Analysis

After extensive profiling, we identified three primary bottlenecks: database connection pooling under load, inefficient JSON serialization in our middleware stack, and suboptimal cache hit ratios on frequently accessed endpoints.

The Solution

We implemented a multi-pronged approach: migrated to connection pooling with PgBouncer, replaced JSON serialization with a faster custom serializer, and introduced a write-through cache layer with Redis.

Results

p99 latency dropped from 85ms to 48ms — a 43% improvement. Median latency improved from 12ms to 7ms. All this with zero downtime during the rollout.

We've published our benchmarking tools and configuration as open source. Check them out on GitHub.