Performance Optimization: Latency, Throughput & Amdahl's Law

Consider an API handler that needs data from both a database and a third-party service before it can respond, where neither call depends on the other's result. Call them one after another and the response waits for the sum of both durations - a 150ms database call plus a 200ms API call is 350ms of latency. But since the two calls are genuinely independent, there's no reason to wait for one before starting the other.

This is the fan-out/fan-in pattern, also called scatter-gather: dispatch every independent slow operation to its own worker thread at the same time, then wait for all of them to finish. Because the I/O waits overlap in wall-clock time, total latency approaches the slowest of the durations rather than their sum. The prerequisite is that the subtasks are genuinely independent - no shared mutable state, and neither needs the other's output as input. If they did, they'd have to run sequentially no matter how many threads are available, which is a small preview of the hard limit Amdahl's Law describes later in this topic.

The same idea applies to CPU-bound work in a different shape. Image processing is a textbook case of data-parallelism: every pixel's transformation is completely independent of every other pixel's, so an array can be split into contiguous, non-overlapping chunks - one per CPU core - with zero coordination between chunks. On an 8-core machine, that's close to an 8x speedup over one thread working through the whole image serially. The sharp difference from the I/O case: for CPU-bound work, adding threads beyond the core count doesn't help - it just adds context-switch overhead, since the bottleneck is compute, not waiting. For I/O-bound work, running far more threads than cores can be productive, because those threads spend most of their time blocked rather than consuming CPU.

package concurrency.performance;

import java.util.concurrent.*;

public class FanOutFanInLatencyDemo {
    public static void main(String[] args) throws Exception {
        // Pool sized to 2 so both independent subtasks run truly concurrently
        // instead of queueing behind each other on a single worker thread.
        ExecutorService executor = Executors.newFixedThreadPool(2);
        long start = System.currentTimeMillis();

        // submit() returns immediately with a Future; the lambda body starts
        // executing on a pool thread right away, in parallel with the next call.
        Future<String> dbResult = executor.submit(() -> {
            Thread.sleep(150);
            return "db-row";
        });
        Future<String> apiResult = executor.submit(() -> {
            Thread.sleep(200);
            return "api-payload";
        });

        // get() blocks only until that specific Future completes. Because both
        // tasks were already running concurrently, the combined wait is close
        // to max(150, 200)ms, not the sum of both.
        String merged = dbResult.get() + " + " + apiResult.get();
        long elapsed = System.currentTimeMillis() - start;

        System.out.println("Merged: " + merged);
        System.out.println("Latency: " + elapsed + "ms (sequential would be ~350ms)");
        executor.shutdown();
    }
}

A naive server design fails under load in two opposite ways. Spawn an unbounded thread per incoming request, and memory runs out eventually - each platform thread reserves a stack of roughly 512KB to 1MB - while excessive context-switching degrades throughput instead of improving it. Cap the thread count but queue incoming requests without any limit instead, and the server keeps accepting work it can never keep up with: memory and latency both grow silently until the process runs out of memory or is effectively dead. This is the classic "queue explosion" failure mode.

Where latency optimization is about one request, throughput optimization is about many concurrent clients, and the fix is bounding every stage of the pipeline: the pool size (both a core size and a maximum), the backlog queue, and what happens once both are full. ThreadPoolExecutor exposes exactly these knobs directly, rather than hiding them behind a convenience factory.

corePoolSize is the minimum number of worker threads kept alive even when idle, ready to pick up work with no startup latency. maximumPoolSize is the cap the pool may grow to - creating extra threads beyond the core size, once the queue is completely full - to absorb bursts. keepAliveTime controls how long those extra threads may sit idle before being reclaimed, so the pool shrinks back down after a burst subsides. The workQueue is the fixed-capacity backlog: too small and the rejection policy fires too eagerly; too large and backpressure only kicks in after memory and latency damage is already done. The rejection handler only fires once both the pool and the queue are saturated; CallerRunsPolicy runs the rejected task on the calling thread itself, which slows down whoever is submitting work and naturally throttles new requests without failing them outright.

The convenience factories most people reach for by default - Executors.newFixedThreadPool(), newCachedThreadPool() - use an unbounded queue internally. That silently reintroduces the exact queue-explosion risk a hand-configured ThreadPoolExecutor exists to prevent: with an unbounded queue, maximumPoolSize becomes unreachable in practice, because the pool never grows past its core size while the queue can always absorb more work.

package concurrency.performance;

import java.util.concurrent.*;

public class BoundedThreadPoolWebServer {
    public static void main(String[] args) {
        // core=10 threads always ready; grows to max=50 under burst load; idle
        // extra threads beyond core are reclaimed after 60s; a bounded queue of
        // 100 caps the backlog so memory/latency can't grow without limit;
        // CallerRunsPolicy self-throttles once pool and queue are both full.
        ExecutorService webServerExecutor = new ThreadPoolExecutor(
            10, 50, 60L, TimeUnit.SECONDS,
            new ArrayBlockingQueue<>(100),
            new ThreadPoolExecutor.CallerRunsPolicy()
        );

        System.out.println("Handling incoming requests...");
        for (int i = 0; i < 15; i++) {
            final int requestId = i;
            webServerExecutor.submit(() ->
                System.out.println("Handling request " + requestId + " on " + Thread.currentThread().getName())
            );
        }
        // Orderly shutdown: no new tasks accepted, but already-queued tasks
        // are still allowed to finish before pool threads terminate.
        webServerExecutor.shutdown();
    }
}

Adding more threads doesn't make a program proportionally faster forever. Some part of almost every program has to run one step at a time - reading input, merging results at the end, a single lock somewhere - and no amount of extra cores can speed up that part. Amdahl's Law makes this intuition precise:

Speedup(N) = 1 / (S + (1 - S) / N)

where S is the fraction of the work that must run sequentially and N is the number of processors. As N grows large, the (1-S)/N term shrinks toward zero, but S itself never does. Speedup asymptotically approaches a hard ceiling of 1 / S, no matter how many cores are thrown at the problem. If 25% of a program is sequential, speedup is capped at 4x, full stop - adding a thousand cores past that point buys nothing.

The practical implication is that performance work should start by finding and shrinking the sequential fraction - a lock held too broadly, an I/O step that can't be split, a single-threaded merge step - before adding threads. At S = 90%, going from 8 to 64 cores buys almost nothing (roughly 2.91x to 3.83x), because the 10% sequential cost dominates everything else. Capacity-planning decisions - "how many cores should we buy" - should be grounded in this ceiling, not in a naive assumption of linear scaling with core count.

package concurrency.performance;

public class AmdahlsLawCalculator {
    // s = sequential fraction (0.0 to 1.0), n = number of processors.
    static double speedup(double s, int n) {
        return 1.0 / (s + (1.0 - s) / n);
    }

    public static void main(String[] args) {
        int[] processors = {1, 2, 4, 8, 16, 32, 64};
        double[] sequentialFractions = {0.0, 0.1, 0.25, 0.5, 0.9};

        System.out.printf("%-8s", "Cores");
        for (double s : sequentialFractions) System.out.printf("  S=%.0f%%  ", s * 100);
        System.out.println();

        for (int n : processors) {
            System.out.printf("%-8d", n);
            for (double s : sequentialFractions) System.out.printf("  %5.2fx  ", speedup(s, n));
            System.out.println();
        }
        // At S=90%, even 64 cores only reaches ~1.10x speedup - the
        // sequential 10% dominates everything else.
    }
}

Real-world Java frameworks make Amdahl's Law and the latency/throughput distinction concrete: Spring Boot's default Tomcat thread pool caps concurrent request handling at a fixed size (throughput-oriented, bounded), while ForkJoinPool (used by parallel streams) is tuned for CPU-bound divide-and-conquer work matched to core count, and virtual threads target exactly the I/O-bound, high-concurrency case where thousands of blocking calls need to overlap without a thread-per-request memory cost.