BEFORE
Buffered ingress and Kafka delivery
- SQS
- Lambda
- Core API
- Outbox + Kafka connector
- Downstream systems
Redis also retained subscription data broadly. The extra services did not solve the API's memory and CPU pressure.
Scaling a newsletter platform while cutting backend cost by nearly 50%
The existing path struggled as more brands and markets came online. I used a measured Core API prototype to make the case for a simpler architecture, then helped turn that decision into an efficient production platform.
Condé Nast · Software Engineer III · High-level technical account
01 / THE CONSTRAINTGrowth exposed
Growth exposed
the cost of complexity.
The platform needed to onboard more markets and brands without multiplying pods and managed-service cost.
~30 RPSold API instability observed
6 podsup to, per region, before redesign
>200 RPSoptimized API prototype in one pod
512 MBprototype pod memory allocation
The original Core API was not optimized for memory or CPU use. It became unstable around 30 requests per second; even with up to six pods running in each region, OOMKilled events still occurred. Redis held the full subscription dataset rather than being limited to repeated reads. SQS, Lambda, and Kafka added operating cost around an API that needed more efficient code.
02 / SYSTEM DESIGNRemove layers after
Remove layers after
proving capacity.
These diagrams show the main data paths at a high level; internal implementation details are omitted.
AFTER
Direct API and lean orchestration
Direct subscription requests and scheduled consent jobs enter the optimized Core API.
- Core API
- Queue collection
- Node.js orchestrator
- Downstream systems
Redis caches repeated database reads. The orchestrator watches the queue collection and passes messages to downstream systems.
SCHEDULED CONSENT PATH
Multiple sources and S3 → scheduled jobs → transformed records → Core API
The scheduler runs about 70 jobs a day, with roughly 30,000 records per job.
03 / DECISIONS & OWNERSHIPAn evidence-led
An evidence-led
architecture change.
Each removed component was backed by a concrete change in application capacity or responsibility.
Start with the API
I built a memory- and CPU-optimized Core API prototype that served more than 200 RPS in one 512 MB pod, then presented the result to the team and director.
Remove unnecessary ingress
That result supported removing SQS and Lambda ahead of the API. I narrowed Redis use to caching repeated database calls instead of retaining all subscription data.
Retire Kafka delivery
I replaced the costly Kafka connector with a Node.js service that watches a queue collection and orchestrates messages to downstream systems in a single 512 MB pod.
Handle market consent jobs
The scheduler pulls consent data from multiple sources and S3. Jobs transform records and submit them to the Core API before the orchestrator passes them downstream.
The tradeoff
The simpler deployment put more of the queue watching and downstream handoff in our Node.js code. We accepted that responsibility because the measured workload fit within a small pod and the new path reduced operating cost.
04 / OUTCOMELower cost,
Lower cost,
room to grow.
Capacity measurements informed the design; the production platform now supports a larger operating footprint.
Nearly 50%lower backend cost from the full architecture overhaul
50M+subscriber records supported
~400 RPSobserved production peak during major events
~180 MBtypical job-runner memory within a 512 MB pod
The >200 RPS figure is from the Core API prototype; the ~400 RPS figure is an observed production peak. The cost reduction reflects the full backend overhaul, including service removal and code optimization.