Synchronous vs Asynchronous: Blocking vs Non-Blocking, and Why It Matters for LLM Apps
~13 min read
Synchronous code waits, doing nothing else, until an operation finishes. Asynchronous code can work on other things while waiting — essential for LLM apps, where a single generation can take many seconds.
Synchronous vs Asynchronous: Blocking vs Non-Blocking, and Why It Matters for LLM Apps is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Synchronous (blocking) code does nothing else while waiting for an operation — simple to reason about, but wastes time that could go to other work
- •Asynchronous (non-blocking) code can work on other tasks while waiting, using async/await in Python, coming back once the awaited result is ready
- •LLM API calls are unusually slow (seconds, not milliseconds), making this distinction matter enormously for AI apps specifically
- •A synchronous server serving LLM requests can only handle one slow request at a time per worker, blocking every other user behind it
- •An asynchronous server can have hundreds of slow LLM requests in flight simultaneously on one worker, which is why modern AI backends are built async from the ground up