Note: This blog post has been done as an independent pricing evaluation for the benefit of Vantage users and the broader infrastructure community. This is not a sponsored post and was not commissioned by either party.
Streaming is eating the world. In particular, event streaming, where event producers, like websites and apps, generate user events 24/7 and feed them to event consumers like recommendation engines, fraud detection systems, and advertising platforms. Apache Kafka is the go-to open-source event streaming project and modern businesses are leveraging its capabilities to respond to customers in real-time. Two options to deploy Kafka at scale are Confluent and Amazon MSK. Each offers advantages and in this post, we will explore how they charge and examples where one may be favorable to the other.
Originally published September 29, 2022.
Amazon MSK Serverless Pricing
In 2022, Amazon released Amazon MSK Serverless which is a much more comparable competitor to Confluent than classic Amazon MSK, which requires users to provision their own servers and manage scaling themselves. That said, MSK is the cheapest option in terms of pure server costs for experienced teams with well-understood workloads and limited feature needs. More on that below.
| Dimension | Price |
|---|---|
| Compute-hours | $0.75 per hour |
| Partition-hours | $0.0015 per partition per hour |
| Storage | $0.10 per GiB-month |
| Data In | $0.10 per GiB |
| Data Out | $0.05 per GiB |
| Connectors | $0.11 per hour |
| Schema Registry | Free via AWS Glue |
Amazon MSK pricing dimensions
Kafka messages are organized in Topics which are distributed across Partitions which are replicated across Brokers which run within Clusters. All this enables Kafka’s legendary high availability. When Kafka clusters are running, charges accumulate for both compute-hours and partition-hours. The number of partitions you create is a function of how much throughput you need for your brokers (basically the speed at which messages should be available). Finally, you can set message retention policies which will eat up storage costs. On MSK, messages can be stored for up to one day. Data in and data out is exactly what you would expect.
Confluent Pricing
There are two public pricing plans—Standard and Enterprise. Some key differences besides price are that Enterprise has higher throughput and private endpoint networking capabilities.
| Dimension | Standard | Enterprise |
|---|---|---|
| Compute-hours | $1.50 per hour | $2.25/E-CKU/hour (minimum 2 E-CKUs) |
| Partition-hours | $0.0015 per hour (first 500 included) | Free |
| Storage | $0.08 per GiB-month | $0.08 per GiB-month |
| Data In/Out (varies by cloud provider, availability zone and networking type) | $0.04 - $0.13 per GiB | $0.05 - $0.11 per GiB |
| Connectors | $0.03 - $3.00 per hour | $0.03 - $3.00 per hour |
| Schema Registry | $0.002 per schema per hour (first 100 included) | $0.002 per schema per hour (first 100 included) |
Confluent pricing dimensions
Confluent prices along exactly the same dimensions as MSK, although we should give them credit for establishing these first. The first thing to note is that Confluent costs more than MSK on a very basic level, but this comes with numerous caveats. First, Confluent offers discounts for longer-term commitments, second Confluent has a “complete data streaming platform” and in our view that is an accurate description. Their feature set is much more comprehensive than MSK, and there is no need to glue (literally in the case of AWS Glue) multiple services together. In particular, here are some major differentiating features from Confluent:
- ksqlDB
- Comprehensive admin panel (query topics from a web browser)
- Local development clusters
- Public access to cluster endpoints
- Managed connectors that need minimal configuration
- Deployment options beyond AWS including GCP and Azure
- The option to add Flink to Confluent in one unified platform
Free Tier MSK vs Confluent
Unlike many AWS services, there is no free tier for MSK. Confluent does have a free tier that gives you access to most of the features of its platform. You must still pay for storage, data in, data out, and partitions at the rates above and there are hard caps of 1 GB/s and 5 TB of storage. You can create up to 4096 partitions for free and compute-hours are also free.
MSK vs Confluent Pricing Scenarios
Normally in our pricing comparisons, we design hypothetical scenarios to demonstrate where one service or the other is cheaper. With the publicly stated pricing here, MSK is always cheaper than Confluent. In fact, if cost is your only concern you can consider using the non-serverless option for MSK and save even more money.
But at what total cost? Kafka is a complicated piece of technology and most businesses choose to offload the scaling, upgrading, and partitioning of it to a managed service built by Kafka experts. The cost to your business, if the message broker that connects your applications goes down, is likely far greater than the savings you realize by running the more bare-bones offering from Amazon. In the end, the best approach is to evaluate the level of Kafka expertise in-house and the consistency of your workload. On one side of the spectrum with less expertise and spikier workloads, Confluent’s breadth of features and support comes out ahead. On the other side, with experienced Java and Kafka engineers and a very predictable workload, you can save money with MSK.
