Avoid S3 Cost Overruns

At the end of April, a blog went viral detailing how an empty S3 bucket suddenly incurred massive charges due to unauthorized incoming requests. While AWS has committed to resolving this particular issue, there are other ways your S3 bill can unexpectedly skyrocket. Fortunately, proactive monitoring and alerting measures can help you avoid getting caught up in a runaway S3 cost horror story.

How One Bucket Was Charged Over $1,300 for Unauthorized S3 Requests

Maciej Pocwierz, a Senior Software Engineer at Semantive, created a standard PoC of a document indexing system for their client. When they checked on it a couple of days later, expecting they were within the free tier, they found the bill racked up to over $1,300. After troubleshooting, they realized a rare issue had occurred, causing them to receive many unauthorized incoming requests, which they had to pay for. In their case, it was because an open-source tool happened to set the default configuration to back up to their bucket.

In this situation, the costs were practically unavoidable. The bucket was empty and private. Requests came directly from the S3 API, so preventative measures like WAF and CloudFront don’t work. Neither does enabling Requester Pays, as the bucket owner is still responsible for request authentication fails. The only solution is to delete the bucket entirely.

Though you can give S3 buckets random names and suffixes to try and obscure them, that’s not 100% effective. Not to mention S3 bucket names weren’t intended to be a security mechanism. Think of all the publicly accessible URLs from hosting static websites or sharing files that contain the name of S3 buckets. That’s why AWS is no longer charging customers.

However, costs can spike in other ways, such as a public bucket getting spammed with requests, or an accidental infinite loop. Setting up a good monitoring and alert system is the best way to catch the problem before it gets too expensive.

Cost Monitoring and Alerts

By utilizing a monitoring platform, you can better understand your costs and receive alerts if unexpected costs arise. Some necessary visibility actions to help you combat unexpected S3 costs before they spiral out of control are:

• Viewing overall costs
• Viewing costs filtered/grouped by bucket, API call, data transfer, etc.
• Viewing anomalies and receiving alerts

AWS has basic built-in tools that you can enable for monitoring and alerts. Alternatively, if you want additional functionalities in one product, you can use a cost monitoring and optimization tool, like Vantage.

See S3 Costs

Having a comprehensive view of your costs allows you to understand where your costs are coming from so you’re aware of spending patterns and can understand anomalies. With Vantage, you can create Cost Reports with unlimited filters and views to see as granular a cost dashboard as you need. For example, you can view S3 costs segmented by bucket, region, storage class, custom tags, and more. You can even see costs broken down by API request type.

To create a Cost Report follow these steps:

1. Navigate to the Cost Reporting tab
2. On the top right of the screen, click New
3. On the top left of the screen, click Filters
4. Configure the following filters:
   • For Provider, select ‘AWS’
   • Select ‘Service’ in New Rule then select ‘Amazon Simple Storage Service’
   • Apply any other filters such as region, storage class, custom tags, etc. for more observability
5. On the top right, click Save as New

You can view by line, bar, area, or pie chart in cumulative, daily, weekly, or monthly views. In addition, you can adjust the date to any previous or forecasted periods and compare periods.

To see a view that can help you notice the spike of unauthorized requests, add these parameters:

1. Select ‘Category’ in New Rule, then select ‘Amazon Simple Storage Service’ is ‘API Request’
2. Above the chart, open the Group By menu. Select Region and Resource.

Vantage Cost Reports S3

This view shows all of your S3 API costs grouped by region and bucket, which in the case of the unauthorized requests would have helped in pinpointing the source of the unauthorized requests, as you can identify the regions and buckets where the spike occurred.

See S3 Costs Terraform

Vantage also has a Terraform provider, allowing you to define and provision Cost Reports as code. This approach ensures consistent reporting across your teams and environments. To see the previous Cost Report with your current month’s costs displayed as a line chart use the following snippet:

terraform {
  required_providers {
    vantage = {
      source = "vantage-sh/vantage"
    }
  }
}

provider "vantage" {
  # this can also be configured with export VANTAGE_API_TOKEN=<YOUR_API_TOKEN>
  # and this block removed entirely:
  api_token = var.api_token
}

resource "vantage_cost_report" "s3_report" {
  title               = "S3 Report"
  filter = <<-FILTER
    (costs.provider = 'aws' AND costs.service = 'Amazon Simple Storage Service') AND
    (costs.provider = 'aws' AND (costs.service = 'Amazon Simple Storage Service' AND costs.category = 'API Request'))
    FILTER
  groupings = "region,resource_id"
  workspace_token = "<YOUR_WORKSPACE_TOKEN>"
  # optionally, use folder_token instead of workspace_token
  # folder_token = "fldr_12345"
}

Vantage Cost Reports S3 using Terraform

S3 Anomalies

Arguably the most powerful feature for catching cost spikes before they become too big of a problem is anomaly detection. Vantage uses an ML model, trained on your Cost Reports, to continuously analyze your spend patterns and create an anomaly when there is a deviation from expected spending patterns. You can receive alerts for any detected cost anomalies by email, Slack, or Teams by following these steps:

1. On the top of any Cost Report, click on the Anomalies toggle
2. On the top right, click Configure Alert
3. Select recipients or if applicable, channels
4. Enter an Alert Threshold

Conclusion

Unexpected S3 costs can be a significant burden on your organization’s cloud spending if left unchecked. As demonstrated in the example of the $1,300 bill from unauthorized requests, S3 costs can quickly spiral out of control due to various factors, ranging from misconfigurations to security vulnerabilities or even third-party integrations.

Fortunately, by implementing a proactive monitoring and alerting strategy, you can stay ahead of potential cost overruns and address issues before they become major problems. Tools like Vantage provide comprehensive cost visibility, anomaly detection, and alerting capabilities tailored specifically for S3 and other cloud services, empowering you to take control of your cloud spend.