Text Classification and Clustering
Applied text classification using zero-shot generative models, embedding+classifier pipelines, and fine-tuned encoders; plus unsupervised text clustering and topic modeling with K-Means, HDBSCAN, and BERTopic — all grounded in embedding space.
Embedding Inputs:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Choose between zero-shot generative classification, embedding+classifier, and fine-tuned encoder approaches based on label count and data availability
- •Build a text classifier by extracting embeddings and training a logistic regression or SVM head
- •Explain why cosine similarity in embedding space enables zero-shot and few-shot classification
- •Run K-Means and HDBSCAN clustering on a document corpus and interpret the clusters
- •Apply BERTopic to discover latent topics in a large unstructured text collection
What is it?
Text classification assigns a category label to a piece of text (positive/negative, spam/ham, topic A/B/C/D). Text clustering groups similar texts together without predefined labels. Topic modeling discovers latent themes in a corpus. All three tasks share a common foundation: representing text as dense embedding vectors, then applying geometric operations (distance, clustering algorithms) in that space.
Jay Alammar (Hands-On Large Language Models, Ch.4/5) organizes the approach spectrum from lightest to heaviest: (1) zero-shot generative classification (prompt a capable LLM, no training), (2) embedding + lightweight classifier (embed text, train a logistic regression or SVM head), (3) fine-tuned encoder (update model weights on your labeled data). Clustering and topic modeling sit orthogonally — they are unsupervised and produce label sets from the data itself rather than applying predefined labels.
Why it exists
Text is unstructured. Databases can't GROUP BY sentiment or JOIN ON topic. Organizations need to automatically route support tickets, moderate content, detect intents in chatbot messages, and discover emerging topics in customer feedback — all at scale, without a human reading every document.
Before embedding models, this required hand-crafted features (TF-IDF, n-gram counts) that couldn't capture semantic meaning. 'Bank account' and 'checking account' share zero words but mean nearly the same thing — TF-IDF treats them as unrelated. Embedding models learn that these are semantically close, enabling classifiers and clustering algorithms to work on meaning rather than surface form.
The generative model wave added a new option: just ask GPT-4 'is this review positive or negative?' — no training required. But this is slow, expensive, and overkill for well-defined, high-volume classification tasks. Understanding when each approach is appropriate is the core skill.
Problem it solves
- I have 100,000 support tickets/day and need to route each to the right team — no human can read them all.
- I have 50 labeled examples of spam/not-spam — not enough to fine-tune a model, but I need a classifier.
- I have 10,000 unlabeled customer reviews — I want to understand what topics customers are discussing.
- I need to classify text into 200 categories — zero-shot generative works for 10 categories but struggles at 200.
- My existing keyword-based classifier misses paraphrases — 'refund my money' routes incorrectly because it doesn't have 'refund' as a keyword.
Intuition
Embeddings turn text into geometric objects — points in a high-dimensional space where semantic similarity becomes physical proximity. Once you have that, classification and clustering become geometry problems:
Classification: 'which labeled region does this new point fall into?' A logistic regression learns a hyperplane that separates classes in embedding space. K-Nearest Neighbors asks 'what label do my nearest neighbors have?'. Zero-shot classification asks 'which class label embedding is this text embedding closest to?' — no training required.
Clustering: 'find groups of points that are densely packed together.' K-Means partitions the space into K Voronoi regions. HDBSCAN finds dense islands of any shape and labels sparse regions as noise. BERTopic runs UMAP to reduce dimensionality, then HDBSCAN to cluster, then TF-IDF on each cluster to name the topics.
Jay Alammar (Ch.4): 'Embeddings turn semantic similarity into a distance function, and distance functions are the input to every clustering and classification algorithm.'
Analogy
Text classification and clustering in embedding space is like sorting a library.
Naive (keyword-based): sort books alphabetically by title. 'Machine Learning' and 'Pattern Recognition' are far apart, even though they cover the same subject. 'The Art of War' and 'The Art of Cooking' are close, even though they're about completely different things.
Embedding-based: place books in a 3D space where books with similar CONTENT are physically close, regardless of title. Now 'Machine Learning' and 'Pattern Recognition' sit next to each other. Classification = deciding which bookshelf region a new book belongs to based on where its nearest neighbors are. Clustering = finding natural clumps of books without any predefined shelf system — the clusters emerge from the content similarity structure of the collection itself.
Technical explanation
TEXT CLASSIFICATION APPROACHES (Jay Alammar, Ch.4):
-
Zero-Shot Generative Classification
- Prompt a generative LLM with the text + label options
- Example: 'Classify this review as [positive, negative, neutral]: {text}'
- Pros: zero training data, works out-of-box for clear categories
- Cons: slow (1-3s/text), expensive ($0.001-0.01/text), unreliable for > 20 classes
- Best for: prototyping, rare labels, ambiguous categories needing reasoning
-
Zero-Shot via Embedding Similarity
- Embed the text and each label string
- Assign label whose embedding is most cosine-similar to text
- Works because semantic meaning is shared: 'positive' is geometrically close to positive reviews in a well-trained embedding space
- Faster and cheaper than generative, but less accurate on nuanced categories
-
Embedding + Classifier Head
- Embed all labeled examples once (one API call per example)
- Train a classical ML classifier on the embedding vectors:
- Logistic Regression: fast, interpretable, good baseline
- SVM (RBF kernel): excellent with 100-10K examples
- XGBoost: strong for imbalanced classes
- At inference: embed new text, run classifier. Embedding model is frozen.
- Pros: cheap inference (no LLM call), fast, works with 100+ examples
- Cons: limited by embedding model quality for very domain-specific text
-
Fine-Tuned Encoder (BERT/RoBERTa)
- Update encoder weights on your labeled data end-to-end
- Needs 1K-10K labeled examples for reliable improvement
- Higher accuracy ceiling but slower to set up and expensive to train
- When to use: high-volume production task with sufficient labeled data
FEW-SHOT CLASSIFICATION WITH EXAMPLES IN CONTEXT: Include 3-5 (text, label) examples in the prompt before the classification target. Improves zero-shot performance significantly, especially for ambiguous categories.
CLUSTERING ALGORITHMS:
K-Means:
- Partitions into exactly K clusters by minimizing intra-cluster distance
- Requires K upfront (use elbow method or silhouette score to choose)
- Assumes spherical, roughly equal-size clusters
- Fast: O(n·K·d) per iteration; practical at millions of points
- Weakness: sensitive to outliers, assumes clusters are convex
HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise):
- Finds clusters of any shape and size
- Does not require K upfront — discovers the number of clusters from data
- Labels low-density points as noise (-1) — excellent for real-world messy corpora
- min_cluster_size parameter: minimum number of points to form a cluster
- Slower than K-Means but far more flexible
BERTOPIC: Full pipeline: SBERT embeddings → UMAP (5 dims) → HDBSCAN → c-TF-IDF per cluster
- c-TF-IDF: TF-IDF computed at the cluster level (treating each cluster as one document)
- Output: each cluster has a topic representation (top-N discriminative words)
- Enables dynamic topic modeling: update topics as new documents arrive
- Can use LLM to name topics: pass cluster's top words to GPT → natural language topic name
Architecture
Production Text Classification System:
┌────────────────────────────────────────────────────────────────┐ │ OFFLINE: Train │ │ │ │ Labeled examples ──► Embedding Model ──► [embed_1, embed_2...]│ │ (100-10K texts) (e.g. 1536-dim) │ │ │ │ │ Classifier Head │ │ (LogReg / SVM / XGB) │ │ │ │ │ Serialize to disk │ └────────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────────┐ │ ONLINE: Inference │ │ │ │ New Text ──► Embedding Model ──► 1536-dim vector │ │ (one text) (cached, shared) │ │ │ Classifier Head │ │ │ │ │ label + confidence score │ │ (< 50ms, no LLM API call) │ └────────────────────────────────────────────────────────────────┘
BERTopic at Scale:
┌────────────────────────────────────────────────────────────────┐ │ Raw corpus (10K-10M docs) │ │ │ │ │ [Sentence Transformer] → N × 768-dim matrix │ │ │ │ │ [UMAP n_components=5] → N × 5-dim (preserves local structure)│ │ │ │ │ [HDBSCAN min_size=10] → cluster_labels: [-1, 0, 0, 1, 2...] │ │ │ (-1 = noise, 0,1,2... = topic IDs) │ │ [c-TF-IDF per cluster] → top-10 discriminative words │ │ │ │ │ [Optional LLM naming] → 'Cloud Computing', 'Customer Billing' │ └────────────────────────────────────────────────────────────────┘
Workflow
FOR CLASSIFICATION:
-
Assess your situation:
- < 10 labels, few examples, prototyping → zero-shot generative
- 100+ labeled examples, need fast inference → embed + classifier
- 1K+ labeled examples, production-critical → fine-tuned encoder
-
Embed+classifier path: a. Embed all labeled examples (one API call per example, batch if large) b. Split: 80% train, 20% test c. Fit LogisticRegression(C=1.0, max_iter=1000) on training embeddings d. Evaluate: accuracy, F1 per class, confusion matrix e. Cache embeddings — avoid re-embedding the same text
-
Handle class imbalance:
- Use class_weight='balanced' in LogisticRegression
- Oversample minority class (SMOTE) in embedding space
- Evaluate with macro-F1, not accuracy (accuracy misleads on imbalanced data)
FOR CLUSTERING/TOPIC MODELING:
- Embed all documents
- Run UMAP(n_components=5, n_neighbors=15, min_dist=0.0) to reduce dimensionality
- Run HDBSCAN(min_cluster_size=10) on UMAP output
- Inspect noise fraction: if > 30% noise, lower min_cluster_size
- For topic names: run c-TF-IDF per cluster, optionally LLM-name top-K words
- Visualize: UMAP 2D scatter colored by cluster label
Example
# Embed + Classifier for text classification import numpy as np from sklearn.linear_model import LogisticRegression from sklearn.metrics import classification_report from anthropic import Anthropic client = Anthropic() def embed_texts(texts: list[str]) -> np.ndarray: # Note: use your embedding provider's API # This example uses a hypothetical embed endpoint. # In practice: openai.embeddings.create() or similar. import openai oc = openai.OpenAI() resp = oc.embeddings.create( model='text-embedding-3-small', input=texts, ) return np.array([e.embedding for e in resp.data]) # 1. Embed labeled training data train_texts = ['I love this!', 'Terrible product', 'Works fine', 'Horrible', 'Great!'] train_labels = ['positive', 'negative', 'positive', 'negative', 'positive'] X_train = embed_texts(train_texts) # shape: (5, 1536) # 2. Train classifier on embeddings clf = LogisticRegression(C=1.0, max_iter=1000, class_weight='balanced') clf.fit(X_train, train_labels) # 3. Inference: embed + classify (no LLM call at inference time) new_texts = ['Absolutely love it', 'Broken on arrival', 'OK I guess'] X_new = embed_texts(new_texts) predictions = clf.predict(X_new) probabilities = clf.predict_proba(X_new) # confidence scores print(list(zip(new_texts, predictions))) # BERTopic clustering (pip install bertopic umap-learn hdbscan) from bertopic import BERTopic from sentence_transformers import SentenceTransformer # Load a sentence transformer for embedding embed_model = SentenceTransformer('all-MiniLM-L6-v2') # Your documents docs = [ 'Cloud computing has transformed enterprise infrastructure.', 'AWS EC2 and S3 are widely used in production.', 'Customer service response times need improvement.', 'Refund processing takes too long.', # ... many more documents ... ] # Fit BERTopic topic_model = BERTopic(embedding_model=embed_model, min_topic_size=3) topics, probs = topic_model.fit_transform(docs) # Get topic info print(topic_model.get_topic_info()) # Output: DataFrame with topic_id, count, top_words per topic
Real-world usage
-
Notion AI (content tagging): runs embedding + logistic regression to classify user documents into topics for their search index. Zero-shot LLM was too slow at scale; the embed+classifier pipeline runs in < 50ms per document.
-
Spotify (podcast topic discovery): uses UMAP + HDBSCAN on podcast episode embeddings to discover topic clusters for their recommendation engine. No predefined topic taxonomy — clusters emerge from the content.
-
Amazon (customer review routing): embeds customer reviews, classifies into product/shipping/service/billing categories for routing to the right team. Embedding approach replaced a 500-keyword rule system that required constant maintenance and still missed paraphrases.
-
Jay Alammar (Hands-On LLMs, Ch.4): 'The embed+classifier approach is the pragmatic workhorse of production NLP. Zero-shot LLMs are great for prototyping and ambiguous tasks, but embedding + logistic regression outperforms them on well-defined classification problems once you have 500+ labeled examples.'
-
BERTopic authors (Grootendorst 2022): used in academic paper topic discovery, customer feedback analysis, and news clustering at organizations including the Allen Institute for AI and various NLP research labs.
Trade-offs
Zero-shot generative vs. embed+classifier: zero-shot requires no labeled data and handles ambiguous, reasoning-heavy categories well; embed+classifier requires 100+ labels but is 10-100x faster and cheaper at inference. For < 5 well-defined classes with zero labeled data, zero-shot; for > 5 classes in production with labeled data, embed+classifier wins.
K-Means vs. HDBSCAN: K-Means is faster and requires no hyperparameter tuning once K is chosen; HDBSCAN finds irregular cluster shapes and handles noise better. For well-separated, roughly spherical clusters, K-Means is simpler. For real-world text where clusters have varying density and shape, HDBSCAN is superior.
UMAP before clustering vs. clustering raw embeddings: raw 768-dim embeddings suffer from the curse of dimensionality — distance metrics become meaningless in very high dimensions. UMAP reduction to 5-50 dims before clustering dramatically improves cluster quality. Trade-off: UMAP is stochastic; results vary across runs (set random_state for reproducibility).
Visual explanation
Classification Approach Comparison:
Data needed Speed Cost Accuracy (typical)
Zero-shot (LLM) 0 examples Slow $$$ Good for ≤10 classes Embed+classifier 100+ examples Fast $ Excellent for ≥5K examples Fine-tuned enc. 1K+ examples Fast $$ Best, needs more labeled data
Embedding + Classifier Pipeline:
Text inputs │ ▼ [Embedding Model] (e.g., text-embedding-3-small) │ produces 1536-dim vector per text ▼ [Classifier Head] (Logistic Regression / SVM / XGBoost) │ trained on labeled embedding vectors ▼ Label predictions + confidence scores
BERTopic Pipeline:
[Raw Documents] │ ▼ [Sentence Transformer] → 768-dim embeddings │ ▼ [UMAP] → 5-dim reduction (preserves local structure, faster clustering) │ ▼ [HDBSCAN] → cluster assignments + noise label (-1) │ ▼ [c-TF-IDF per cluster] → top terms that distinguish each cluster │ ▼ Topics: ['machine learning', 'data science'] ['customer service', 'refund']
Zero-Shot Classification via Embedding Similarity:
Label embeddings: embed('positive') → v_pos embed('negative') → v_neg Text embedding: embed('This product is amazing!') → v_text Decision: argmax(cosine_sim(v_text, v_pos), cosine_sim(v_text, v_neg)) → 'positive' (v_text is closer to v_pos in embedding space)
Advantages
- —
Embed+classifier inference requires no LLM API call — < 50ms latency, < $0.0001/text cost at scale
- —
Embedding-based approaches naturally handle paraphrases — 'refund my order' and 'return my purchase' both map to similar embedding regions
- —
BERTopic discovers topics from data without a predefined taxonomy — useful for exploratory analysis of new domains
- —
Zero-shot classification requires zero labeled data — can bootstrap a classifier in minutes for prototyping
- —
The classifier head is swappable without re-embedding — try logistic regression, SVM, and XGBoost on the same frozen embeddings
Disadvantages
- —
Embed+classifier quality is bounded by the embedding model — if the embedding model doesn't capture domain-specific semantics, the classifier ceiling is low
- —
BERTopic cluster quality is highly sensitive to UMAP and HDBSCAN hyperparameters — requires tuning and visual inspection, not a fully automated pipeline
- —
Zero-shot classification performance degrades quickly as the number of classes increases beyond 20-30
- —
HDBSCAN labels a significant fraction of points as noise (-1) in sparse datasets — may not produce complete coverage
- —
Clustering produces unlabeled clusters that require manual interpretation — discovering that Cluster 7 is about 'billing issues' still requires a human to name it
Common mistakes
- —
Using accuracy as the metric for imbalanced classification. If 90% of examples are 'not spam', a classifier that always predicts 'not spam' achieves 90% accuracy. Use macro-averaged F1, precision-recall curves, and per-class F1. Always report class distribution alongside accuracy.
- —
Not caching embeddings. Each embedding API call costs money and time. If you embed the same text twice (e.g., once during training and once at inference for a cached example), you've wasted both. Cache embeddings keyed by text hash. For large corpora, store embeddings in a vector database.
- —
Clustering in full embedding dimensionality. HDBSCAN and K-Means degrade in high dimensions (curse of dimensionality). Always apply UMAP or PCA to reduce to 5-50 dimensions before clustering. Jay Alammar (Ch.5): 'Never run K-Means on 1536-dim OpenAI embeddings directly — reduce to 50 dimensions first.'
- —
Setting K in K-Means without validation. Choosing K=10 'because it sounds right' produces poor clusters. Use the elbow method (plot inertia vs. K) or silhouette score (higher is better) to choose K. For exploratory work, prefer HDBSCAN which discovers K from the data.
- —
Interpreting all BERTopic clusters as meaningful. HDBSCAN's noise label (-1) exists because not everything clusters cleanly. If 40% of your documents are noise, the remaining clusters are meaningful but incomplete. Don't report cluster coverage as 100% if 40% of documents were unclassified.
🎤 Interview questions
Compare zero-shot generative classification, embedding+classifier, and fine-tuned encoder approaches. When would you choose each, and what data and infrastructure do they each require?
Explain how BERTopic discovers topics. What does each stage (UMAP, HDBSCAN, c-TF-IDF) contribute, and why is UMAP needed before HDBSCAN?