Common Tasks: Image Captioning, Visual QA, Document Understanding, and OCR
~12 min read
Four task categories cover most real VLM use cases — describing an image, answering questions about it, understanding structured documents, and reading text within an image — each with different difficulty and reliability profiles.
Common Tasks: Image Captioning, Visual QA, Document Understanding, and OCR is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Image captioning generates an open-ended description of an image — the least demanding task, useful for accessibility and content indexing
- •Visual Question Answering (VQA) requires attending to a SPECIFIC detail a question asks about, not just a generically plausible description
- •Document understanding extends VQA to structured, information-dense inputs (tables, charts, forms), requiring accurate parsing of precisely-positioned information
- •OCR (reading literal text) is distinct from document understanding (interpreting structure/meaning) — a model can nail one while failing the other
- •Reliability generally decreases from captioning to VQA to document understanding/OCR on messy real documents, which is why production systems often add verification steps for the harder tasks