Data Lake Architecture on AWS
Combining S3, Glue, Athena, and Lake Formation into a single architecture pattern for storing and analyzing data at any scale, in any format.
Want a visual for this topic?
Generate a diagram tailored to Data Lake Architecture on AWS — the AI picks whichever visual (architecture, flowchart, ER diagram, etc.) best fits this specific AWS concept.
Sign in to generate a visual →🎓 Learning objectives
- •Explain what a data lake is and how it differs from a traditional data warehouse
- •Explain the role of AWS Lake Formation in a data lake architecture
- •Describe the layered (raw/curated/analytics-ready) organization common in data lake design
What is it?
A data lake is a centralized repository (on AWS, almost always built on S3) that stores data at any scale, in its original raw format, without requiring you to define a rigid structure upfront — unlike a traditional data warehouse, which requires data to be transformed into a predefined schema before it's loaded. AWS Lake Formation is a service that simplifies setting up, securing, and managing a data lake, building on top of S3, Glue, and fine-grained access control.
Why it exists
Traditional data warehouses require deciding on a schema before loading data (schema-on-write), which works well for well-understood, structured data but is restrictive for the messy variety of modern data — logs, JSON events, images, sensor data — where forcing a rigid upfront schema is impractical or loses information. Data lakes exist to store this raw, varied data cheaply and flexibly (schema-on-read — you decide the structure only when you actually query it), while still being organized and governed well enough to be genuinely useful rather than becoming an unmanageable 'data swamp.'
Problem it solves
It solves the schema-rigidity problem (store data in its native format now, decide how to interpret and structure it later, per query, rather than upfront), the storage-cost problem (S3's low cost per GB makes storing large volumes of raw, unprocessed data economically practical, unlike a traditional warehouse's typically higher storage cost), and the governance problem specifically for Lake Formation (centralized, fine-grained access control across many different data sources feeding into the lake, rather than each data source having its own separate, inconsistent access model).
Intuition
A traditional data warehouse is like a library that only accepts books already catalogued and shelved in a specific, predefined system — very efficient to search once a book is in, but restrictive about what can be added and how. A data lake is more like a large, organized storage facility that accepts items in whatever form they arrive, cataloging and structuring them for retrieval as needed, rather than requiring every item to be reformatted into a single rigid structure before it's even allowed in.
Analogy
A data lake is like a well-organized recycling and sorting facility that accepts materials in whatever form they arrive (raw data) and sorts/processes them into useful streams as needed for different purposes downstream, versus a traditional factory that only accepts pre-processed raw materials meeting an exact specification (a data warehouse's schema-on-write requirement) before they're allowed onto the production line at all.
Technical explanation
AWS Lake Formation builds on Glue's Data Catalog and S3, adding centralized permission management that can be applied at a much finer grain (specific columns, or rows matching a filter) than S3 bucket policies alone could practically express — critical when different consumers of the same underlying data lake should see different subsets (e.g. a finance team seeing full customer records, a marketing team seeing only anonymized aggregate data from the same underlying tables). The layered zone pattern (raw → curated → analytics-ready) is a design convention, not an AWS-enforced structure, but is widely adopted because it cleanly separates 'data exactly as it arrived, for reprocessing/audit purposes' from 'data that's been validated and is safe/ready for actual business consumption,' preventing accidental use of unvalidated raw data in a business-critical report.
Architecture
A retail company's data lake ingests raw point-of-sale transaction data, web clickstream events, and inventory system exports into an S3 Raw zone, unmodified from their source formats. Scheduled Glue ETL jobs clean, deduplicate, and convert this data into a Curated zone in partitioned Parquet format. A further aggregation job produces daily sales summaries in an Analytics-Ready zone, which QuickSight dashboards and ad-hoc Athena queries consume directly — with Lake Formation ensuring the marketing team can only see aggregated, non-PII customer data while the finance team has broader access to the full underlying transaction records, all governed centrally rather than through inconsistent, separately-managed S3 bucket policies per team.
Workflow
- Land raw data from various sources into an S3 'raw' zone in its original format, unmodified. 2) Use Glue Crawlers/ETL jobs to clean, standardize, and convert this data into a 'curated' zone, typically in a columnar format like Parquet with sensible partitioning. 3) Build further aggregated, business-logic-specific datasets in an 'analytics-ready' zone for direct consumption. 4) Use Lake Formation to define fine-grained, centrally-managed access permissions across the lake's tables, rather than relying purely on IAM/S3-bucket-level access control. 5) Expose the lake to consumers via Athena for ad-hoc queries, Redshift Spectrum for integration with an existing warehouse, or QuickSight for dashboards.
Example
A healthcare research organization's data lake ingests raw clinical data feeds from many different hospital systems into a Raw zone, standardizes them into a common schema in a Curated zone, and uses Lake Formation's fine-grained, row-level access control to ensure researchers can only query de-identified patient records relevant to their specific approved study, from an underlying dataset that also contains fully identified records only a small, separately-authorized compliance team can access — a level of access precision that would be difficult to achieve cleanly with S3 bucket policies alone.
Real-world usage
The S3-plus-Glue-plus-Athena data lake pattern is AWS's standard reference architecture for big data analytics, extensively documented across AWS's analytics whitepapers; Lake Formation is commonly adopted specifically once an organization's data lake has grown to serve multiple teams with genuinely different, fine-grained access requirements over the same underlying data.
Trade-offs
A data lake's schema-on-read flexibility trades upfront rigor (a traditional warehouse's schema-on-write validation catches format problems immediately at load time) for flexibility in accommodating varied, evolving, or not-yet-fully-understood data — appropriate when data variety and volume are high and requirements are still evolving; a traditional warehouse's stricter upfront structure remains preferable for well-understood, stable, structured reporting needs where that upfront rigor's benefit outweighs the flexibility cost.
Visual explanation
Picture S3 organized into layered zones: a Raw zone holding data exactly as it arrived from source systems, unmodified; a Curated (or Cleaned) zone holding that same data after basic cleaning, deduplication, and format standardization (often converted to Parquet); and an Analytics-Ready (or Consumption) zone holding further aggregated, business-logic-applied data ready for direct consumption by BI tools or specific applications. Glue Crawlers and ETL jobs move and transform data between these zones, with Lake Formation providing fine-grained, centrally-managed access control (down to the column or row level) across all of it.
Advantages
- —
Schema-on-read flexibility accommodates varied, evolving data types without requiring upfront rigid structure
- —
S3's low storage cost makes retaining large volumes of raw historical data economically practical
- —
Lake Formation's fine-grained access control enables safely sharing one underlying data lake across teams with genuinely different access needs
- —
The layered zone pattern cleanly separates raw/unvalidated data from curated, business-ready data, reducing the risk of using unvalidated data in critical decisions
Disadvantages
- —
Without deliberate governance (cataloging, access control, the layered zone pattern), a data lake can devolve into an unmanageable, undocumented 'data swamp' that's hard to actually use
- —
Schema-on-read flexibility means query-time errors from inconsistent or unexpected data formats are more common than a traditional warehouse's upfront schema validation would catch
- —
Building and maintaining the ETL pipelines moving data through raw/curated/analytics-ready zones requires real ongoing engineering investment
- —
Fine-grained Lake Formation permissions add real design and configuration complexity beyond simpler, coarser-grained S3 bucket policies
Common mistakes
- —
Building a data lake without any governance or organizational structure (no zones, no cataloging discipline), resulting in an unmanageable 'data swamp' that's technically searchable but practically unusable
- —
Relying purely on IAM/S3 bucket policies for access control on a multi-team data lake instead of Lake Formation's fine-grained permissions, when different teams genuinely need different, more precise views of the same underlying data
- —
Allowing business-critical reports or dashboards to query the raw zone directly instead of the validated, curated/analytics-ready zones, risking unvalidated or inconsistent data driving real business decisions
- —
Underinvesting in the ETL pipeline connecting raw to curated data, letting the curated zone silently fall out of sync with new data landing in the raw zone
In the AWS Console
- 1
AWS Console → Lake Formation → Data lake locations → Register location
Register your S3 bucket(s)/prefixes as data lake locations under Lake Formation's management.
This is the step that brings existing S3-based data under Lake Formation's centralized permission model rather than relying purely on IAM/bucket policies.
- 2
Lake Formation → Permissions → Data lake permissions → Grant
Grant specific, fine-grained permissions (e.g. table-level, or column-level for tables that support it) to specific IAM principals or roles.
This centralizes access control decisions in one place rather than spreading them across many separate S3 bucket policies and IAM policies.
🎤 Interview questions
What's the fundamental difference between a data lake and a traditional data warehouse? (Listen for: data lake uses schema-on-read (structure decided at query time, flexible, varied raw formats); data warehouse uses schema-on-write (structure defined and validated upfront, more rigid but more immediately consistent).)
What does AWS Lake Formation add on top of raw S3 and Glue? (Listen for: centralized, fine-grained access control (down to column/row level in some cases) across the data lake, simplifying and unifying permission management beyond what S3 bucket policies alone provide.)
Why do data lakes commonly use a layered (raw/curated/analytics-ready) zone structure? (Listen for: separates unvalidated raw data from cleaned, validated data, preventing accidental use of unvalidated data in business-critical reporting while preserving raw data for reprocessing/audit.)
What's the risk of building a data lake without deliberate governance? (Listen for: becomes an unmanageable, undocumented 'data swamp' — technically stores everything but practically hard to discover, trust, or use effectively.)