For Amazon Managed Streaming for Apache Kafka (Amazon MSK), express brokers—a new broker type—are now widely accessible. It is designed to reduce recovery time by 90%, scale up to 20 times faster, and give up to three times the throughput per broker as compared to typical brokers running Apache Kafka. Express brokers provide the same low latency performance that Amazon MSK clients are used to, support Kafka APIs, and come preconfigured with Kafka best practices by default. This lets customers continue using their existing client applications without any changes.
Express brokers enable improved compute and storage elasticity for Kafka applications when using clusters provided by Amazon MSK. Amazon MSK, a fully managed AWS service, can facilitate the development and operation of scalable and highly available Apache Kafka-based systems.
Let's take a closer look at some of the primary traits and benefits of Express brokers:
- Simple operations and hands-free storage management: Express brokers remove disk-related constraints by providing infinite storage without preprovisioning. Cluster sizing is made simpler by dividing the recommended per-broker throughput by the ingress and egress throughput alone. This eliminates the need for proactive disk capacity monitoring and scaling, increasing resilience and simplifying cluster administration by eliminating a potential point of failure.
- Decreased brokers with up to three times each throughput Higher throughput per broker allows smaller clusters to manage the same demand. The throughput of m7g.16xl standard brokers must take client traffic and background processes into consideration, even if they can easily accommodate 154 MBps ingress. Because to resource isolation and brokers' opinionated settings, m7g.16xl size instances may safely manage up to 500 MBps ingress during cluster events without compromising availability or performance.
- 20 times faster scaling and higher utilization: Express brokers may scale up to 20 times faster than Standard brokers by lowering data transfer. Cluster resizing becomes more reliable and faster as a result. The ability to deploy brokers in a couple of minutes and monitor each broker's ingress throughput capacity eliminates the need for over-provisioning in anticipation of traffic surges.
- 90% faster recovery and increased resilience: Express brokers are designed for mission-critical applications that require a high degree of resilience. They are preconfigured with best-practice defaults, like 3-way replication (RF=3), to reduce failures caused by misconfiguration. Additionally, express brokers recover from momentary failures 90% faster than standard Apache Kafka brokers. Express brokers' recovery and rebalancing, which use comparatively less cluster resources, facilitate capacity planning. This eliminates the need for continuous monitoring and the potential for increased resource consumption when clusters are sized appropriately.
| MSK provisioned | MSK Serverless | ||
| Standard brokers | Express brokers | ||
| Configuration range | Most flexible | Flexible | Least flexible |
| Cluster rebalancing | Customer managed | Customer managed but up to 20x faster | MSK managed |
| Capacity management | Yes | Yes (compute only) | No |
| Storage management | Yes | No | No |
How to start using brokers for Amazon MSK Express
The Sizing and Pricing worksheet from Amazon MSK can be used to begin working with Express brokers. This worksheet helps you figure out the cluster size needed to sustain your workload and gives you an approximate monthly cost estimate.
The throughput requirements of your workload are the primary factor that determines the size of your cluster. You should also consider other factors, like as the number of connections and the partition, to estimate the size and number of brokers your cluster will need. For example, if your streaming application needs 30 MBps of data ingress (write) and 80 MBps of data egress (read) capacity, you can use three express.m7g.large brokers to meet your throughput requirements (assuming the workload's partition count is within the maximum number of partitions that Amazon MSK recommends for a m7g.large instance).
The maximum entrance, egress, and partition counts per instance size that are advised for long-term, secure operations are displayed in the following table. Further details on these recommendations can be found in the Best Practices section of the Amazon MSK Developer Guide.
Instance size Ingress (MBps) Egress (MBps)
express.m7g.large 15.6 31.2
express.m7g.4xlarge 124.9 249.8
express.m7g.16xlarge 500.0 1000.0
Once you know how many Express brokers you need for your workload, you can use the CreateCluster API or the AWS Management Console to create an Amazon MSK provisioned cluster.
When establishing a new cluster in the Amazon MSK panel, select Express brokers under Broker type. After that, decide how much processing power you want to give the broker. Express brokers can use Graviton-based systems with Apache Kafka 3.6.0 installed, as the screenshot illustrates. Pre-provisioning storage is not necessary for express brokers.
To further tailor the performance of your clusters to your preferences, you can also change some of these configurations.
An MSK cluster is created using the AWS CLI create-cluster command.
JSON file brokernodegroupinfo. For Amazon MSK to distribute broker nodes in JSON, you define three subnets.
You can connect your clients to the cluster endpoints using the bootstrap connection string after the cluster has been formed.
You can scale horizontally (by adding brokers) or vertically (by altering the instance size) with Express brokers. Throughput is doubled by vertical scaling without the need to redistribute partitions. You can create additional partitions and add brokers in groups of three thanks to horizontal scalability, but new brokers handling traffic necessitate partition reassignment.
The capacity of Express Brokers to add brokers and rebalance partitions in a matter of minutes is one of its main features. It could take hours to rebalance partitions after adding Standard brokers. When three Express brokers are added to a cluster and 2000 partitions are redistributed to each of the additional brokers, the graph below illustrates how long it takes to rebalance partitions.
How long does it take to rebalance partitions when a cluster with 2000 partitions and three Express brokers is added?
As demonstrated, it takes roughly ten minutes to reassign partitions and use the new brokers' capacity. The same experiment was conducted on a comparable cluster of Standard brokers, and partition reassignment occurred over a 24-hour period.
Considerations
You should be aware of the following regarding Express brokers:- Data migration: Your Kafka or MSK cluster's data is transferred to an Express broker cluster by Amazon MSK Replicator. as well as metadata from one cluster to another.
- Monitoring: Keep an eye on your Express broker cluster both internally and at the broker level by using Amazon CloudWatch metrics. Prometheus can also be utilized with the JMX Exporter and Node Exporter to enable open monitoring and reveal metrics.
- Security: Similar to other broker types, Amazon MSK offers transparent server-side encryption for the storage in Express brokers through integration with AWS Key Management Service (AWS KMS). When you build up an MSK cluster with Express brokers, you can provide Amazon MSK with the AWS KMS key to encrypt your data. It is not moving. Amazon MSK generates an AWS managed key and uses it in your place if you don't supply a KMS key.
Currently accessible
Currently, the following regions provide the Express broker type: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm).The hourly fee for Express brokers is determined by the amount and resolution of Apache Kafka broker instances (invoiced at one-second resolution). The hourly rate is also based on how many brokers are active in your MSK clusters. Additionally, there is a cost per gigabyte (GB) for data written to an Express broker.
0 Comments