Demystifying What’s Causing AWS Config Costs

Thousands of changes can happen in your AWS account every day. Keeping track of them all is where AWS Config comes in. But this observability comes with a small cost. In normal operation, this is de minimis, but when resources are misconfigured, it can add up quickly.

In this article, we’ll go over what can happen to make Config costs skyrocket as well as some techniques to debug costs and prevent unexpected spikes.

What is AWS Config?

AWS Config is a service used to view and evaluate the configurations of resources within your AWS account. One way to think of it is as an audit trail of anything that happens within your account. Every time a monitored resource changes a configuration item is recorded. Resources in AWS are a wide range of entities you work with like as EC2 instances, S3 buckets, or databases.

Once you have the configuration items you can apply Config rules to them. Config rules are essentially predefined criteria or policies that Config uses to assess whether your resources comply with specific guidelines. For instance, you might have a Config rule that checks whether all your S3 buckets are encrypted. Some use cases include resource administration, auditing and compliance, managing and troubleshooting configuration changes, and security analysis.

AWS Config Pricing and Free Tier

Config pricing is based on several factors:

• Configuration Item Recordings: The rate is $0.003 for each configuration item recorded.
• Config Rule Evaluations:

• Conformance Pack Evaluations (when a resource is evaluated by a Config rule within a Conformance Pack):

Note - there may be additional charges for S3 storage of snapshots and history files, SNS charges for change notifications, and Lambda charges if you create custom rules.

Why Is This Confusing?

We have noticed a few folks publicly complaining about Config cost overruns:

• Chris Farris
• Simon Berens
• Borys Serebrov
• Cloudsoft
• Corey Quinn

In the list above, Serebrov suddenly had their Config costs increase from $5 to $100 per month.

The cause of cost spikes for many of these cases are things like services stuck in restart loops that go unnoticed but are recorded as huge amounts of Config items. A user within our customer base encountered a situation where an ECS Cluster became stuck in a continuous cycle of deleting and recreating its two ENIs hundreds of times each hour. This unexpected behavior resulted in a significant increase in the number of AWS Config items being generated. That can add up at $0.003 an item, especially if there are additional rule evaluations.

The confusion can be largely attributed to a lack of visibility. To determine pricing you need to know what resources generate Config rules.

How to See Config Costs

An example of a Config cost report with an anomaly.

Amazon CloudWatch is a tool that can be used for Config to improve visibility and validate your setup. As a monitoring service, CloudWatch specializes in the collection and tracking of metrics, enabling you to create custom dashboards, search and analyze data, and set up notifications. Some examples of recorded metrics include the “number of CIs recorded by resource type, the number of failed configuration history exports to your Amazon S3 buckets, and the number of times AWS Config recording received permissions errors from AWS Identity Access and Management (IAM).” Still, in the case of a sudden cost spike, additional research is needed to determine where and why the anomaly arises. For that, you can use Amazon Athena.

Athena helps to improve visibility into your Config costs and resource changes and helps you troubleshoot cost anomalies. With Athena, you can query your Config data and create custom reports, providing insights into what resources are generating Config items and driving costs. To set up the Athena integration follow these steps from AWS.

In the case of wanting to identify what’s causing a sudden surge, you can follow the steps and query the “count of the CIs, per resource type and resource ID, for your desired time period.”

A query like this

SELECT configurationItem.resourceType,
       configurationItem.resourceId,
       COUNT(configurationItem.resourceId) AS NumberOfChanges
FROM default.awsconfig
CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
WHERE "$path" LIKE '%ConfigHistory%'
      AND configurationItem.configurationItemCaptureTime >= '2020-02-01T%'
      AND configurationItem.configurationItemCaptureTime <= '2020-02-1T%'
GROUP BY configurationItem.resourceType,
         configurationItem.resourceId
ORDER BY NumberOfChanges DESC

Example query from AWS

will return something like this.

resourceType              resourceId        NumberOfChanges
AWS::EC2::VPC             vpc-12345678        3304
AWS::EC2::Subnet          subnet-23456789     2568
AWS::EC2::SecurityGroup   sg-34567890         1986

Example result returned from the AWS query

Using the result you may already be able to understand the surge.

However, to investigate further you can use AWS CloudTrail to view resource IDs and see what entity is changing them. CloudTrail is useful for tracking and logging AWS account activity so you can see who, when, and from which location the changes are being made. In this case, you can use CloudTrail logs to troubleshoot the issue using the resource IDs from Athena.

Debugging and Optimizing Config Costs

Okay, so you’ve figured out what can cause spikes in costs and how to view them. There are also preventative measures you can take. AWS provides several tips for Config cost optimizations. Some of the most helpful are:

1. Select Only Required Resources: When configuring Config you have the option of selecting from all resources supported in the region, (with the additional option of including global resources) or specific resource types. Be sure to only select the resources needed.
2. Select Only Required Region: Choose to monitor resources only in the regions that are necessary for your specific use case.
3. Lifecycle Policy for Configuration History: Reduce storage costs by auto-deleting configuration history records after a specified number of days.
4. Avoid Conformance Pack Duplicates: Some Conformance Packs contain the same rules, make sure to customize them to avoid being charged twice.
5. Create Billing Alerts: Set up billing alerts using AWS services like CloudWatch or AWS Budgets. These alerts notify you when your Config costs approach or exceed predefined thresholds, allowing for proactive cost monitoring.
6. AWS Cost Anomaly Detection: Cost Anomaly Detection can be used to monitor your Config costs for unusual spending patterns or unexpected cost spikes.

Conclusion

Config is a useful tool for essential tasks such as motoring configuration items and maintaining compliance. However, high costs and unexpected cost surges can occur, making the tool challenging to manage. By using Athena to analyze your Config data and implementing other cost optimization recommendations you can ensure there are no surprises. That way you can gain visibility and ensure your resources remain compliant and well-monitored.