Easily build complex reports
Monitoring and efficiency metrics
Custom cost allocation tags
Network cost visibility
Organizational cost hierarchies
Budgeting and budget alerts
Discover active resources
Consumption-based insights
Alerts for unexpected charges
Automated AWS cost savings
Discover cost savings
Unified view of AWS discounts
COGS and business metrics
Model savings plans
Collaborate on cost initiatives
Create and manage your teams
Automate cloud infrastructure
Cloud cost issue tracking
Detect cost spikes
by Emily Dunenfeld
Contents
Thousands of changes can happen in your AWS account every day. Keeping track of them all is where AWS Config comes in. But this observability comes with a small cost. In normal operation, this is de minimis, but when resources are misconfigured, it can add up quickly.
In this article, we’ll go over what can happen to make Config costs skyrocket as well as some techniques to debug costs and prevent unexpected spikes.
AWS Config is a service used to view and evaluate the configurations of resources within your AWS account. One way to think of it is as an audit trail of anything that happens within your account. Every time a monitored resource changes a configuration item is recorded. Resources in AWS are a wide range of entities you work with like as EC2 instances, S3 buckets, or databases.
Once you have the configuration items you can apply Config rules to them. Config rules are essentially predefined criteria or policies that Config uses to assess whether your resources comply with specific guidelines. For instance, you might have a Config rule that checks whether all your S3 buckets are encrypted. Some use cases include resource administration, auditing and compliance, managing and troubleshooting configuration changes, and security analysis.
Config pricing is based on several factors:
Note - there may be additional charges for S3 storage of snapshots and history files, SNS charges for change notifications, and Lambda charges if you create custom rules.
We have noticed a few folks publicly complaining about Config cost overruns:
In the list above, Serebrov suddenly had their Config costs increase from $5 to $100 per month.
The cause of cost spikes for many of these cases are things like services stuck in restart loops that go unnoticed but are recorded as huge amounts of Config items. A user within our customer base encountered a situation where an ECS Cluster became stuck in a continuous cycle of deleting and recreating its two ENIs hundreds of times each hour. This unexpected behavior resulted in a significant increase in the number of AWS Config items being generated. That can add up at $0.003 an item, especially if there are additional rule evaluations.
The confusion can be largely attributed to a lack of visibility. To determine pricing you need to know what resources generate Config rules.
An example of a Config cost report with an anomaly.
Amazon CloudWatch is a tool that can be used for Config to improve visibility and validate your setup. As a monitoring service, CloudWatch specializes in the collection and tracking of metrics, enabling you to create custom dashboards, search and analyze data, and set up notifications. Some examples of recorded metrics include the “number of CIs recorded by resource type, the number of failed configuration history exports to your Amazon S3 buckets, and the number of times AWS Config recording received permissions errors from AWS Identity Access and Management (IAM).” Still, in the case of a sudden cost spike, additional research is needed to determine where and why the anomaly arises. For that, you can use Amazon Athena.
Athena helps to improve visibility into your Config costs and resource changes and helps you troubleshoot cost anomalies. With Athena, you can query your Config data and create custom reports, providing insights into what resources are generating Config items and driving costs. To set up the Athena integration follow these steps from AWS.
In the case of wanting to identify what’s causing a sudden surge, you can follow the steps and query the “count of the CIs, per resource type and resource ID, for your desired time period.”
A query like this
SELECT configurationItem.resourceType, configurationItem.resourceId, COUNT(configurationItem.resourceId) AS NumberOfChanges FROM default.awsconfig CROSS JOIN UNNEST(configurationitems) AS t(configurationItem) WHERE "$path" LIKE '%ConfigHistory%' AND configurationItem.configurationItemCaptureTime >= '2020-02-01T%' AND configurationItem.configurationItemCaptureTime <= '2020-02-1T%' GROUP BY configurationItem.resourceType, configurationItem.resourceId ORDER BY NumberOfChanges DESC
Example query from AWS
will return something like this.
resourceType resourceId NumberOfChanges AWS::EC2::VPC vpc-12345678 3304 AWS::EC2::Subnet subnet-23456789 2568 AWS::EC2::SecurityGroup sg-34567890 1986
Example result returned from the AWS query
Using the result you may already be able to understand the surge.
However, to investigate further you can use AWS CloudTrail to view resource IDs and see what entity is changing them. CloudTrail is useful for tracking and logging AWS account activity so you can see who, when, and from which location the changes are being made. In this case, you can use CloudTrail logs to troubleshoot the issue using the resource IDs from Athena.
Okay, so you’ve figured out what can cause spikes in costs and how to view them. There are also preventative measures you can take. AWS provides several tips for Config cost optimizations. Some of the most helpful are:
Config is a useful tool for essential tasks such as motoring configuration items and maintaining compliance. However, high costs and unexpected cost surges can occur, making the tool challenging to manage. By using Athena to analyze your Config data and implementing other cost optimization recommendations you can ensure there are no surprises. That way you can gain visibility and ensure your resources remain compliant and well-monitored.
Concerned aboutcloud costs?
See your AWS savingswithin 60 seconds.
Grafana is a strong competitor to the monitoring and observability features of Datadog for a fraction of the price.
AWS is implementing a policy update that will no longer allow Reserved Instances and Savings Plans to be shared across end customers.
Use FinOps as Code to explore all your active cost-generating provider resources.