Easily build complex reports
Monitoring and efficiency metrics
Custom cost allocation tags
Network cost visibility
Organizational cost hierarchies
Budgeting and budget alerts
Discover active resources
Consumption-based insights
Alerts for unexpected charges
Automated AWS cost savings
Discover cost savings
Unified view of AWS discounts
COGS and business metrics
Model savings plans
Collaborate on cost initiatives
Create and manage your teams
Automate cloud infrastructure
Cloud cost issue tracking
Detect cost spikes
by Emily Dunenfeld
Contents
At the end of April, a blog went viral detailing how an empty S3 bucket suddenly incurred massive charges due to unauthorized incoming requests. While AWS has committed to resolving this particular issue, there are other ways your S3 bill can unexpectedly skyrocket. Fortunately, proactive monitoring and alerting measures can help you avoid getting caught up in a runaway S3 cost horror story.
Maciej Pocwierz, a Senior Software Engineer at Semantive, created a standard PoC of a document indexing system for their client. When they checked on it a couple of days later, expecting they were within the free tier, they found the bill racked up to over $1,300. After troubleshooting, they realized a rare issue had occurred, causing them to receive many unauthorized incoming requests, which they had to pay for. In their case, it was because an open-source tool happened to set the default configuration to back up to their bucket.
In this situation, the costs were practically unavoidable. The bucket was empty and private. Requests came directly from the S3 API, so preventative measures like WAF and CloudFront don’t work. Neither does enabling Requester Pays, as the bucket owner is still responsible for request authentication fails. The only solution is to delete the bucket entirely.
Though you can give S3 buckets random names and suffixes to try and obscure them, that’s not 100% effective. Not to mention S3 bucket names weren’t intended to be a security mechanism. Think of all the publicly accessible URLs from hosting static websites or sharing files that contain the name of S3 buckets. That’s why AWS is no longer charging customers.
However, costs can spike in other ways, such as a public bucket getting spammed with requests, or an accidental infinite loop. Setting up a good monitoring and alert system is the best way to catch the problem before it gets too expensive.
By utilizing a monitoring platform, you can better understand your costs and receive alerts if unexpected costs arise. Some necessary visibility actions to help you combat unexpected S3 costs before they spiral out of control are:
AWS has basic built-in tools that you can enable for monitoring and alerts. Alternatively, if you want additional functionalities in one product, you can use a cost monitoring and optimization tool, like Vantage.
Having a comprehensive view of your costs allows you to understand where your costs are coming from so you’re aware of spending patterns and can understand anomalies. With Vantage, you can create Cost Reports with unlimited filters and views to see as granular a cost dashboard as you need. For example, you can view S3 costs segmented by bucket, region, storage class, custom tags, and more. You can even see costs broken down by API request type.
To create a Cost Report follow these steps:
You can view by line, bar, area, or pie chart in cumulative, daily, weekly, or monthly views. In addition, you can adjust the date to any previous or forecasted periods and compare periods.
To see a view that can help you notice the spike of unauthorized requests, add these parameters:
Vantage Cost Reports S3
This view shows all of your S3 API costs grouped by region and bucket, which in the case of the unauthorized requests would have helped in pinpointing the source of the unauthorized requests, as you can identify the regions and buckets where the spike occurred.
Vantage also has a Terraform provider, allowing you to define and provision Cost Reports as code. This approach ensures consistent reporting across your teams and environments. To see the previous Cost Report with your current month’s costs displayed as a line chart use the following snippet:
terraform { required_providers { vantage = { source = "vantage-sh/vantage" } } } provider "vantage" { # this can also be configured with export VANTAGE_API_TOKEN=<YOUR_API_TOKEN> # and this block removed entirely: api_token = var.api_token } resource "vantage_cost_report" "s3_report" { title = "S3 Report" filter = <<-FILTER (costs.provider = 'aws' AND costs.service = 'Amazon Simple Storage Service') AND (costs.provider = 'aws' AND (costs.service = 'Amazon Simple Storage Service' AND costs.category = 'API Request')) FILTER groupings = "region,resource_id" workspace_token = "<YOUR_WORKSPACE_TOKEN>" # optionally, use folder_token instead of workspace_token # folder_token = "fldr_12345" }
Vantage Cost Reports S3 using Terraform
Arguably the most powerful feature for catching cost spikes before they become too big of a problem is anomaly detection. Vantage uses an ML model, trained on your Cost Reports, to continuously analyze your spend patterns and create an anomaly when there is a deviation from expected spending patterns. You can receive alerts for any detected cost anomalies by email, Slack, or Teams by following these steps:
Unexpected S3 costs can be a significant burden on your organization’s cloud spending if left unchecked. As demonstrated in the example of the $1,300 bill from unauthorized requests, S3 costs can quickly spiral out of control due to various factors, ranging from misconfigurations to security vulnerabilities or even third-party integrations.
Fortunately, by implementing a proactive monitoring and alerting strategy, you can stay ahead of potential cost overruns and address issues before they become major problems. Tools like Vantage provide comprehensive cost visibility, anomaly detection, and alerting capabilities tailored specifically for S3 and other cloud services, empowering you to take control of your cloud spend.
Monitor your AWS costs.
Grafana is a strong competitor to the monitoring and observability features of Datadog for a fraction of the price.
AWS is implementing a policy update that will no longer allow Reserved Instances and Savings Plans to be shared across end customers.
Use FinOps as Code to explore all your active cost-generating provider resources.