advanced~10h

Model Optimization: Compression & KV Cache

Learn quantization, pruning, distillation, and the size and structure of KV Caches.

Subtopics

Exercises

Projects

Quiz Qs

Flashcards

▶📚 Prerequisites(1)

🎓 Learning objectives

•Compare PTQ and QAT quantization techniques
•Explain KV Caching and calculate KV cache size: $O(layers imes heads imes seq imes dim)$
•Describe Multi-Query Attention (MQA) and Grouped-Query Attention (GQA)

Quantization: INT8, INT4, GPTQ and AWQ

KV Cache Optimization and PagedAttention

Batching Strategies: Static, Dynamic and Continuous

Model Pruning and Distillation

llm-foundationsllm-deployment

llm-deployment →