It started with a routine deployment. A new data pipeline went live on a Thursday afternoon. By Monday morning, the AWS bill had jumped $38,000.
No one noticed over the weekend. The dashboard showed a spike, but no one was checking the dashboard. The alert was set to fire at the monthly budget threshold — which wouldn't trigger until week three. By the time anyone looked, four days of runaway spend had already landed.
This isn't a rare story. It's the most common story in cloud cost management.
What Happened
The pipeline was processing event data from a new integration. During testing, it handled a few thousand events per hour. In production, with real traffic, it processed 200,000 events per hour. Each event triggered a Lambda invocation, wrote to DynamoDB, and pushed to an S3 bucket.
The Lambda was provisioned at 1GB memory. DynamoDB was on-demand pricing. S3 was standard storage with frequent PUTs.
None of these are expensive individually. But 200,000 events per hour, 24 hours a day, for four days? The numbers compound fast.
- Lambda: ~19 million invocations over the weekend. Cost: ~$7,600.
- DynamoDB: On-demand writes at scale. Cost: ~$14,200.
- S3 PUT requests: 19 million objects. Cost: ~$9,500.
- Data transfer and misc: ~$6,700.
Total: $38,000. On a service that was budgeted at $2,000/month.
Why No One Caught It
Three reasons:
1. The alert threshold was monthly. The team had a budget alert set at $15,000/month for the account. The spike happened over a weekend, and the monthly total hadn't crossed $15,000 yet on Saturday.
2. No one checks dashboards on weekends. AWS Cost Explorer had the data. But dashboards don't send messages. They wait for you to visit.
3. The deployment seemed normal. CI passed. No errors in logs. The service was working correctly — it was just working at a scale no one anticipated.
What We'd Do Differently
If you could rewind the clock, three things would have caught this within hours instead of days:
Daily anomaly detection. Compare today's spend to the 7-day average. On Friday, the first full day after deployment, spend was 6x the baseline. An anomaly alert would have fired by Saturday morning.
Provider-level daily signals. A simple "green/amber/red" signal per AWS account, delivered to Slack. Even a quick glance at "Red — AWS spend 500% above normal" would have prompted investigation.
Pace-to-forecast alerts. "At current pace, this account will spend $95,000 this month. Budget is $15,000." That message on Saturday morning would have triggered an immediate response.
The Math of Early Detection
The pipeline burned roughly $9,500 per day. If caught on Friday evening instead of Monday morning, the total overspend would have been ~$9,500 instead of $38,000. That's $28,500 saved by detecting the problem one day earlier.
If caught Saturday morning with a daily anomaly alert, the savings would have been ~$19,000. Two days of prevention.
Early detection doesn't eliminate the mistake. It limits the blast radius.
The Lesson
The $38,000 bill wasn't caused by carelessness. The team did everything right — tested the service, deployed through CI, monitored logs. The problem was that cost monitoring wasn't part of the feedback loop.
Logs tell you if something is broken. Cost signals tell you if something is expensive. You need both.
The fix isn't better dashboards or stricter budgets. It's faster feedback. A daily signal that says "something changed" — delivered where your team already works — catches these problems while they're still small.
Every surprise cloud bill has a version of this story. The details change, but the pattern doesn't: a change happens, costs spike, and no one notices until the invoice arrives. The teams that avoid this pattern aren't the ones with the best dashboards. They're the ones with the fastest signals.
Get started: Connect AWS to StackSpend for daily anomaly detection and pace-to-forecast alerts. Learn more about AWS cost monitoring.