Kinesis Data Streams
Real-time, ordered, replayable data streaming at massive scale — for when you need many consumers reading the same continuous flow of events, not just one-time delivery.
Want a visual for this topic?
Generate a diagram tailored to Kinesis Data Streams — the AI picks whichever visual (architecture, flowchart, ER diagram, etc.) best fits this specific AWS concept.
Sign in to generate a visual →🎓 Learning objectives
- •Explain how Kinesis differs fundamentally from SQS
- •Explain shards and how they determine a stream's throughput capacity
- •Explain why multiple independent consumers can all read the same data from a Kinesis stream
- •Recognize when a workload needs Kinesis versus SQS/SNS
What is it?
Amazon Kinesis Data Streams is a real-time data streaming service for continuously ingesting, storing, and processing large volumes of ordered data — clickstreams, IoT sensor readings, application logs, financial transactions — where multiple independent consumers may need to read and process the same flowing data, potentially replaying recent history, not just receive it once.
Why it exists
SQS and SNS are built around the model of 'deliver each message and it's gone (or acknowledged) shortly after.' Some workloads need something fundamentally different: a continuously flowing, ordered log of events that multiple independent applications can each read through at their own pace, potentially including replaying recent data if a consumer needs to reprocess it (e.g. after fixing a bug and wanting to reprocess the last hour of events). Kinesis exists to serve exactly this streaming, replayable, multi-consumer data model.
Problem it solves
It solves the multi-independent-consumer problem (unlike SQS where one message goes to one consumer, many Kinesis consumers can each independently read the entire stream at their own pace without affecting each other), the ordering-at-scale problem (strict ordering is maintained within a shard, even at very high throughput), and the replay problem (data remains in the stream for a configurable retention period — up to 365 days — so a consumer can reprocess recent history, something a standard queue's 'delete after processing' model doesn't support).
Intuition
SQS is like a single mail slot where each letter is taken by one recipient and then it's gone. Kinesis is more like a continuously running news ticker on a building: many different people can each be watching that same ticker at their own pace, all seeing the same sequence of headlines, and if you look away and come back, you can potentially scroll back through recent history — no single viewer 'consumes' a headline in a way that removes it for everyone else.
Analogy
A live sports broadcast versus a phone call: a phone call (SQS) is between exactly two parties, and once the conversation happens, it's over. A broadcast (Kinesis) is watched simultaneously by many independent viewers, each experiencing the same feed at their own screen, and (with a DVR/replay capability) can rewind to catch something they missed, without affecting any other viewer's experience.
Technical explanation
A Kinesis stream's throughput capacity is determined by its number of Shards — each shard supports up to 1MB/sec or 1,000 records/sec of write throughput, and up to 2MB/sec of read throughput (shared among consumers, or higher per-consumer with Enhanced Fan-Out, which gives each registered consumer its own dedicated 2MB/sec pipe rather than sharing). Records are distributed across shards based on a Partition Key (similar in concept to DynamoDB's partition key) — records with the same partition key always land in the same shard, preserving their relative order; records with different keys may be spread across different shards for parallelism. The Kinesis Client Library (KCL) handles the complexity of tracking consumer position (checkpointing) and coordinating multiple worker instances reading from the shards of one stream.
Architecture
A ride-sharing company streams every location ping from every active driver into a Kinesis stream, partitioned by driver_id (ensuring one driver's pings stay in order within a shard). Three independent consumer applications read the same stream simultaneously: a real-time map dashboard, a surge-pricing calculation engine, and a data pipeline loading records into a data warehouse for historical analysis — none of these three consumers interferes with or slows down the others, and if the surge-pricing engine needs to be restarted after a bug fix, it can resume from its last checkpoint or replay recent data, entirely independent of the other two consumers' progress.
Workflow
- Determine if your workload genuinely needs Kinesis's model — multiple independent consumers reading the same ordered data stream, or the need to replay recent history — versus SQS/SNS's simpler deliver-once model. 2) Choose a partition key that groups records needing relative ordering together while still spreading load reasonably across shards. 3) Size the number of shards based on expected write throughput (records/sec and bytes/sec). 4) Use the Kinesis Client Library (or a managed consumer like Lambda, Kinesis Data Firehose, or Kinesis Data Analytics) to build consumer applications, letting the library handle checkpointing and shard coordination.
Example
A fintech company streams every trade execution into Kinesis, partitioned by account_id to guarantee one account's trades stay strictly ordered. A real-time risk-monitoring consumer processes the stream to flag suspicious patterns within seconds, while a completely separate, independently-paced consumer batches records into S3 via Kinesis Data Firehose for long-term storage and later analysis — both consumers read the identical stream of trade events without any coordination or interference between them.
Real-world usage
Kinesis underlies many real-time analytics and monitoring pipelines across AWS customers; Netflix has documented using Kinesis-style streaming architectures for real-time monitoring of viewing activity across their platform; the pattern of Kinesis feeding both a real-time processing path and a batch storage path (often via Kinesis Data Firehose to S3) simultaneously is a standard reference architecture in AWS's own streaming data guidance.
Trade-offs
Kinesis's power (multi-consumer replay, strict ordering at scale) comes with real operational complexity that isn't justified for simple messaging needs — SQS/SNS remain the right, simpler default for point-to-point work distribution or straightforward fan-out. Kinesis becomes the right tool specifically when you have multiple genuinely independent consumers needing to process the same ordered data stream, or a real need to replay/reprocess recent history, which neither SQS nor SNS support natively.
Visual explanation
Picture a Kinesis stream as a continuously growing, ordered log divided into multiple Shards (parallel lanes), each holding an ordered sequence of records. Multiple independent Consumer Applications (e.g. a real-time dashboard, a fraud-detection system, and a data warehouse loader) each maintain their own position (a checkpoint) as they read through the stream at their own pace — none of them removes records for the others, and each can be reading from a different point in recent history simultaneously.
Advantages
- —
Multiple independent consumers can each read the full stream at their own pace with no interference between them, unlike SQS's single-consumer-per-message model
- —
Configurable data retention (up to 365 days) allows replaying recent history, useful for reprocessing after a bug fix or backfilling a new consumer
- —
Strict ordering is maintained within a shard even at very high throughput
- —
Scales to very high ingest throughput by adding shards, suited to high-volume clickstream, IoT, or log data
Disadvantages
- —
Meaningfully more complex to operate and reason about than SQS/SNS — shard management, checkpointing, and consumer coordination all add real operational surface
- —
Choosing shard count and partition key strategy requires understanding your throughput and ordering needs upfront, and resharding an existing stream is a non-trivial operation
- —
Overkill for a simple point-to-point or fan-out messaging need that SQS or SNS would handle far more simply
- —
Consumers must implement or rely on a library (KCL) for proper checkpointing — naive custom consumer code can easily reprocess or skip records incorrectly
Common mistakes
- —
Choosing Kinesis by default for a workload that's actually a simple point-to-point or fan-out messaging need, taking on unnecessary shard-management and checkpointing complexity that SQS/SNS wouldn't have required
- —
Choosing a partition key with poor cardinality, concentrating throughput on a small number of shards ('hot shards') while others sit underutilized, similar to DynamoDB's hot partition problem
- —
Writing custom consumer logic without the Kinesis Client Library's checkpointing support, risking incorrectly skipped or duplicated record processing across worker restarts
- —
Under-provisioning shard count for actual write throughput, causing ProvisionedThroughputExceeded errors and requiring a live resharding operation to fix
In the AWS Console
- 1
AWS Console → Kinesis → Data streams → Create data stream
Choose 'On-demand' capacity mode (Kinesis handles shard scaling automatically) or 'Provisioned' mode (specify the initial shard count based on expected throughput).
On-demand mode removes the need to manually plan and manage shard count, similar in spirit to DynamoDB's On-Demand mode — a reasonable default unless you have specific reasons to manage shards yourself.
- 2
Kinesis → [your stream] → Data viewer / Consumers tab
Register a consumer application, optionally enabling Enhanced Fan-Out for that consumer to get its own dedicated 2MB/sec read throughput rather than sharing with other consumers.
Enhanced Fan-Out has an additional cost per consumer-hour — only enable it for consumers that genuinely need guaranteed, non-shared read throughput.
- 3
AWS Console → Kinesis → Data Firehose → Create delivery stream
Configure Kinesis Data Firehose as a managed consumer of your Data Stream, delivering records automatically to a destination like S3, Redshift, or OpenSearch with no custom consumer code needed.
Firehose is the easiest way to get data from a Kinesis stream into long-term storage without writing and operating your own consumer application.
🎤 Interview questions
How does Kinesis Data Streams fundamentally differ from SQS? (Listen for: SQS delivers each message to one consumer and removes it; Kinesis lets multiple independent consumers each read the full ordered stream at their own pace, with configurable replay/retention.)
What determines a Kinesis stream's throughput capacity, and how do you scale it? (Listen for: the number of shards, each supporting a fixed write/read throughput; scale by adding shards (resharding) or using On-Demand mode for automatic scaling.)
Why might you choose a poor partition key strategy hurt a Kinesis stream's performance? (Listen for: records with the same key always land on the same shard; a low-cardinality or skewed key creates a 'hot shard' bottleneck even if total stream capacity is sufficient.)
When would you choose Kinesis over SNS/SQS for an event-driven system? (Listen for: when multiple genuinely independent consumers need to process the same ordered data stream, or when replaying/reprocessing recent history is a real requirement — not for simple point-to-point or fan-out needs.)
What role does the Kinesis Client Library play for a consumer application? (Listen for: handles checkpointing (tracking consumer position) and coordinating multiple worker instances across a stream's shards, preventing incorrect record skipping or duplication.)