How Multiple Independent Consumers Work
~8 min read
Why reading from Kinesis doesn't remove data, unlike SQS.
Unlike SQS, where receiving and deleting a message removes it from the queue for everyone, reading from a Kinesis stream doesn't remove or affect the data at all — a consumer simply reads records starting from wherever it currently is positioned (tracked via a Sequence Number and checkpoint), and multiple completely independent consumer applications can each maintain their own separate position within the same stream simultaneously.
This is what enables genuinely independent, parallel processing pipelines reading the identical data — a real-time dashboard consumer might be reading records from 2 seconds ago, while a batch analytics consumer might be catching up on records from an hour ago after being offline for maintenance, with neither consumer's activity affecting the other's ability to read any record within the stream's retention window.
Checkpointing (typically handled by the Kinesis Client Library, which stores each consumer application's progress in a DynamoDB table) is what allows a consumer to resume exactly where it left off after a restart, rather than either reprocessing the entire stream from the beginning or losing track of its position entirely.
💬 Deep Dive with AI
Key points
- •Reading from Kinesis doesn't remove data — unlike SQS's delete-after-consume model
- •Each independent consumer tracks its own position via a sequence number and checkpoint
- •Multiple consumers can be at completely different points in the stream simultaneously with no interference
- •The Kinesis Client Library typically manages checkpointing (often backed by a DynamoDB table) so consumers resume correctly after a restart