expertReal System Design Problems
Design Google's Web Indexing Pipeline — petabyte-scale crawl, parse, and index.
The petabyte-scale data-pipeline problem: how the web is crawled at scale, how HTML is parsed and content extracted, how an inverted index is built and continuously updated across hundreds of machines, and how index freshness is maintained. Expect to reason about a MapReduce-style (or equivalent) batch processing pipeline, not a single-machine design.
This is a Pro question
Sign in, then upgrade to Pro or Power to unlock this question and the full Interview Prep bank.
Design Google's Web Indexing Pipeline — petabyte-scale crawl, parse, and index.