Synchronous vs Asynchronous: Blocking vs Non-Blocking, and Why It Matters for LLM Apps

Once you understand that an HTTP request goes through several stages (previous subtopic) — including real network round trips that take actual time — a crucial question follows: what does your program DO while it's waiting for that response to come back? The answer splits into two fundamentally different approaches.

Synchronous (blocking) code waits, doing absolutely nothing else, until the current operation completes, before moving on to the next line. If you call an API and that call takes 3 seconds, your program is frozen — not processing anything else, not responding to any other event — for those entire 3 seconds. This is simple to reason about (code runs top-to-bottom, in order, exactly as written) but wastes an enormous amount of time: your program (or, in a server handling multiple users, your server) could have been doing OTHER useful work during that wait instead of sitting idle.

Asynchronous (non-blocking) code can start an operation, and instead of freezing until it completes, move on to do OTHER work while that operation happens in the background — then get notified (or come back and check) once the result is ready. In Python, this is what the async/await keywords enable: an async function can await a slow operation (like a network call), and while it's waiting, the program's event loop is free to work on other pending tasks — other users' requests, other API calls, other background work — instead of sitting idle the whole time.

Why this matters ENORMOUSLY for LLM apps specifically: LLM API calls are unusually slow compared to most API calls you might be used to — a complex generation can take many seconds, sometimes tens of seconds, rather than the milliseconds a typical database lookup takes. A SYNCHRONOUS server handling LLM requests can only serve ONE user at a time per worker — if user A's request takes 20 seconds, every other user is stuck waiting behind them, even though the SERVER itself isn't doing any real work during those 20 seconds (it's just waiting on the LLM provider). An ASYNCHRONOUS server can have hundreds of these slow LLM requests all 'in flight' simultaneously on a single worker, because during each one's wait, the server is free to accept and start handling other users' requests. This exact reasoning is why modern AI-app backends (like a FastAPI server, covered in the next subtopic) are built async from the ground up — it's the difference between a server that can handle a handful of concurrent users versus hundreds.

# Comparing synchronous (blocking) vs asynchronous (non-blocking)
# handling of 3 slow 'LLM calls' -- notice the total time difference.
import asyncio
import time

def sync_llm_call(name: str, delay: float) -> str:
    """Synchronous: the whole program freezes for `delay` seconds."""
    time.sleep(delay)
    return f"{name} done"

async def async_llm_call(name: str, delay: float) -> str:
    """Asynchronous: awaiting sleep() frees the event loop to do
    other work (like starting the OTHER calls) during the wait."""
    await asyncio.sleep(delay)
    return f"{name} done"

def run_synchronous():
    start = time.perf_counter()
    results = [sync_llm_call(f"request_{i}", 0.3) for i in range(3)]
    print(f"sync total time:  {time.perf_counter() - start:.2f}s -> {results}")

async def run_asynchronous():
    start = time.perf_counter()
    # All 3 'requests' run concurrently -- the event loop juggles them
    results = await asyncio.gather(*[async_llm_call(f"request_{i}", 0.3) for i in range(3)])
    print(f"async total time: {time.perf_counter() - start:.2f}s -> {results}")

run_synchronous()          # takes ~0.9s: 0.3s x 3, one after another
asyncio.run(run_asynchronous())   # takes ~0.3s: all 3 waited on simultaneously