Today, Vantage announces the launch of Network Flow Reports. By combining information about IP traffic on the network with underlying cost data, Vantage customers can now visualize and measure the costs of individual flows within their network, grouped by source and destination at the network, interface, or AWS service level. Network Flow Reports makes it possible to break down and attribute data transfer charges to both specific AWS resources and services outside of AWS (e.g., see the transit costs out to other providers, such as Datadog).
A Network Flow Report filtered to cross-AZ traffic
Before, Vantage was only able to show network costs as attributed in the AWS Cost and Usage Report (CUR). This meant that for each service or resource incurring costs, you could analyze costs up to a certain billing granularity, such as “EU-DataTransfer-Out-Bytes” or “DataTransfer-Regional-Bytes”. While this was helpful for understanding at a high level which resources or AWS services were driving network costs, you were unable to get visibility into what was actually happening on a network level to ultimately drive those costs.
Now, Vantage customers who use AWS can view specific network flows and associated costs for resources that drive costs. From the Active Resources section of the console, you can now create a Network Flow Report. From this report, you can filter to specific sources—such as VPC, Availability Zone, subnet, or specific interfaces—and see and filter on details about their destinations. This view includes a Sankey diagram of the traffic that shows the estimated cost and bytes of each flow. You can drill into the metadata for any resource included in the flow, but you can also get resource-specific network flows from both the Active Resources and Cost Report views in Vantage.
This feature is now available for all Vantage customers. To get started, enable the VPC Flow Log integration in the console. After the integration is enabled, Vantage will begin importing the flows nightly. If you do not yet have VPC Flow Logs enabled in your AWS account, see the AWS documentation. For more information on Network Flow Reports, see the Vantage documentation.
Frequently Asked Questions
1. What is being launched today?
Today, Vantage is launching Network Flow Reports for AWS that allow users to view more details about resources that are accruing network costs. Network Flow Reports provide a detailed and customizable view of the sources and destinations of the network traffic. This feature makes it possible to correlate transfer charges that originate from a specific VPC, instance, or network interface, that traverse a specific network resource, such as an Internet Gateway, NAT Gateway, or PrivateLink, or are destined for external services, such as Datadog.
2. Who can use Network Flow Reports?
The customer is any Vantage user who incurs network transfer costs on AWS and wants to see a more detailed breakdown of these costs. Customers are required to have VPC Flow Logs enabled for any VPC they want to analyze within Vantage.
3. How much do Network Flow Reports cost to use?
There is no additional cost to use this feature; it is included as part of your Vantage subscription. That being said, enabling AWS VPC Flow Logs will incur associated CloudWatch and S3 storage and processing fees as part of your AWS bill. For more information on VPC Flow Logs pricing, see this Cloud Cost Handbook article.
4. How does Vantage integrate with VPC Flow Logs?
Vantage integrates with VPC Flow Logs by granting the Vantage-owned AWS IAM role access to an S3 bucket that stores your VPC Flow Logs. In the event that you already have an existing S3 bucket containing VPC Flow Logs, Vantage will automatically detect and list the buckets available for integration.
The integration must be completed for each AWS account that owns an S3 bucket containing VPC Flow Logs. You can complete this integration via the AWS CLI, AWS Management Console, or the Vantage Terraform provider.
5. I am a customer with existing VPC Flow Logs; can I use them with Vantage?
Yes, Vantage will integrate directly with your existing VPC Flow Logs, and they will be automatically detected during the integration.
6. Can Vantage plug into my existing Amazon CloudWatch destination for VPC Flow Logs?
No. We have chosen not to integrate with Amazon CloudWatch because of the cost of querying CloudWatch. The Vantage integration relies on the files being available in S3.
7. Can I enable this feature for only a subset of my VPCs?
Yes, when you select the S3 buckets for integration, you can select specific VPC Flow Logs within those buckets.
8. Do VPC Flow Logs need to be in a specific version or configuration?
Vantage recommends that you have at least the following fields in your VPC Flow Log format to increase the discoverability of network-related costs.
${action} ${bytes} ${dstaddr} ${start} ${end} ${flow-direction} ${log-status} ${region} ${srcaddr} ${account-id} ${instance-id} ${interface-id} ${subnet-id} ${vpc-id} ${az-id}
A lack of these fields may result in Vantage not being able to properly correlate the network flows to estimated costs.
9. How quickly are VPC Flow Logs generated after enabling them on AWS?
Vantage ingests these logs nightly and will make them available within 24 hours of initially enabling the integration. Before the data is processed for the first time, the integration will indicate whether it can successfully access the corresponding S3 bucket. You can view this status in the Vantage console.
10. What is the supported list of cloud infrastructure providers?
At this time, only AWS is supported. We do plan to support Azure and GCP in the future, as they also support flow logs.
11. What AWS resources can I see network flows for?
When possible, Vantage will correlate the ENI (Elastic Network Interface) Identifier to the corresponding resource, such as a NAT Gateway or an RDS Instance. This correlation is done by syncing the metadata for currently active resources. When available, the Network Flow Report will link to these resources directly.
When you view data transfer costs for a specific resource on a Cost Report, it will also link directly to a filtered Network Flow Report.
12. What fields can I filter on in Network Flow Reports?
By default, a Network Flow Report will display the following fields:
- Source Resource UUID
- Peer Resource UUID
- Traffic Category
You can add the following fields which can also be used for filtering:
- Source Subnet
- Source VPC
- Source Instance ID
- Source Service
- Destination Service
Additional fields are also available. See the documentation for a list of all filtering options.
13. Where do I enable VPC Flow Logs in Vantage?
Navigate to the AWS integration page and select the VPC Flow Logs tab. You can then add an integration. A list of S3 buckets is displayed along with their corresponding VPC Flow Logs. For each unique AWS account where a bucket is located, you will need to grant the corresponding Vantage IAM role access.
AWS Integration screen with VPC Flow Logs onboarding option
14. What are the components of the network flow diagram?
The diagram splits the costs generated by network data transfer across any groupings selected for the Network Flow Report. The available groupings map to the available fields within a particular VPC Flow Log.
15. How can I track the cost of enabling VPC Flow Logs?
These will be represented in the costs for the S3 bucket where your flow logs are written to, as well as an S3-Egress fee from CloudWatch.
16. Is there network traffic that will not be available in these flows?
Yes, Vantage will not import any REJECT flows nor any flows that do not generate corresponding costs.
17. How will external destinations be represented on the diagram?
When possible, Vantage will map external IP addresses to services using a static IP range, such as Datadog or Sentry. If you notice an external IP not being mapped to a service, you can reach out to support@vantage.sh with the range and service. External services are required to publish static IP address lists to be definitively mapped by Vantage.
18. How is the estimated cost calculated?
The estimated cost is calculated by applying your blended data transfer rates to the flow of traffic. For instance, if a flow is going from one subnet to another and those subnets are in different Availability Zones, Vantage will apply your cross-AZ data transfer rate to those bytes. Because data transfer rates can be tiered, and the metadata associated with the destination may move around, the calculation is best effort. However, this will still be useful to identify cost hotspots within your network. See the documentation for more details on this calculation.
19. When viewing a new Network Flow Report, what is the default view?
By default, Vantage displays the top 10 sources and their top destinations by cost.
20. How far back can I query for network flows?
By default, Vantage ingests 7 days of network flows into your account and keeps the data available for 31 days. For enterprise customers, this retention period can be adjusted.
21. Does Vantage surface any cost recommendations based on this data?
Not at this time, but we do plan to surface cost recommendations in the future.
22. Can I be notified when the network flow data is available within Vantage?
After enabling the VPC Flow Log data, you will receive an email when data is ready for the first time.
23. Can I backfill VPC Flow Logs?
Data is only generated going forward after VPC Flow Logs are enabled; however, if you are integrating with an existing flow log, Vantage will automatically import 7 days of data. Reach out to support@vantage.sh if you need to increase this limit.
24. What are some example workflows for Network Flow Reports?
After your data is imported, Vantage provides three pre-delivered Network Flow Reports on the Network Flow Reports screen to help you get started. These reports help with some common scenarios or workflows for viewing reports.
- All Network Flow Logs: This report shows all your network flows. The flow’s Resource UUID and Peer Resource UUID are provided as well as the traffic category (e.g., public). Review this report to get a high-level view of all your traffic flows.
- Cross-AZ Traffic: Cross-AZ data transfer within AWS incurs higher costs compared to intra-AZ data transfer. This report shows the source and destination for resources so you can understand where cross-AZ traffic is originating and going. Use this information to make decisions on whether this traffic can be diverted to stay within the same AZ.
- Public Traffic Destinations: View this report to see all traffic that is going out to the internet. This report can be useful for security reviews and understanding the exact resources that are going out to the internet.
See the documentation for additional details on how to use these reports.
