AWS increases big data efficiency with larger record support, allowing smoother processing for IoT, CDC, and advanced analytics.

Amazon Web Services (AWS) in a recent announcement has announced that Amazon Kinesis Data Streams will now be supporting record sizes up to 10MiB, which is a tenfold increase from the previous limit. With this all-new launch, users can now publish larger data payloads on their data streams, using the existing Kinesis Data Streams APIs in their applications without extra effort.

Amazon, with this launch, is accompanied by a 2x increase in its maximum PutRecords request size from 5MiB to 10MiB. This shift will now simplify the data pipelines and reduce operational overhead for IoT analytics, change the generative AI workloads, and increase the data capture rate. Also, this shift will improve the ease of use and reduce complications while handling data payloads across industries and use cases.

In the current IoT and data analytics domain, many use connected vehicles and industrial equipment, which generate increased volumes of sensor telemetry data, and the size of telemetry records occasionally used to exceed the previous 1MiB limit in Kinesis. This used to add extra burden on the customers, such as splitting large records into multiple smaller datasets or storing the large reports separately and sending only metadata through Kinesis.

But now, with the increased Kinesis record size limit from 1MiB to 10MiB, the need for these types of complex workarounds is cut off, thus reducing operational overhead for customers in IoT, CDC, and advanced analytics use cases. Customers can now ingest and process these intermittent large data records with ease and enhanced security using the same familiar Kinesis APIs.