LLM Evaluation
Metrics, methodology, and pipelines for evaluating language model quality.
Requires LLM Foundations. Statistical background helpful.
LM Evaluation Metrics: Entropy, Cross-Entropy & Perplexity
intermediate~4hThe information-theoretic foundation of language model evaluation: entropy, cross-entropy, perplexity, and bits-per-character — and why these metrics are both powerful and limited.
AI-as-Judge: Using Models to Evaluate Model Outputs
intermediate~4hHow to use capable LLMs as automated evaluators of other model outputs — methodology, bias types, pairwise vs. pointwise patterns, and when AI judging is (and isn't) reliable.
Evaluation Pipeline Design
advanced~5hHow to design a complete LLM evaluation pipeline: decomposing complex tasks into per-component evaluations, writing evaluation guidelines, choosing annotation methods and data, and wiring the pipeline into a CI/CD deployment gate.
Model Selection and Benchmarking
intermediate~4hHow to select the right LLM for your application: navigating public benchmarks, the build-vs-buy decision, and the proprietary vs. open-weight vs. open-source model framework covering cost, control, data privacy, and customization trade-offs.