Clustering and Topic Modeling with Embeddings

Clustering finds structure in unlabeled text corpora. Unlike classification, the label set is discovered from the data.

K-Means Clustering

from sklearn.cluster import KMeans
import numpy as np

# Reduce dimensions first (crucial for high-dim embeddings)
import umap
reducer = umap.UMAP(n_components=50, random_state=42)
reduced = reducer.fit_transform(embeddings)  # (N, 50)

# Choose K using silhouette score
from sklearn.metrics import silhouette_score
scores = {}
for k in range(5, 30, 5):
    km = KMeans(n_clusters=k, random_state=42).fit(reduced)
    scores[k] = silhouette_score(reduced, km.labels_)
best_k = max(scores, key=scores.get)

• Strengths: fast, predictable (exactly K clusters), interpretable centroids
• Weaknesses: requires K, assumes spherical clusters, sensitive to outliers

HDBSCAN

import hdbscan
clusterer = hdbscan.HDBSCAN(min_cluster_size=10, metric='euclidean')
labels = clusterer.fit_predict(reduced_5d)  # -1 = noise
n_topics = len(set(labels)) - (1 if -1 in labels else 0)
noise_pct = (labels == -1).mean() * 100

• Strengths: no K needed, handles irregular shapes, labels outliers as noise
• Weaknesses: slower, noise fraction can be high, sensitive to min_cluster_size

BERTopic Combines embedding → UMAP → HDBSCAN → c-TF-IDF:

• UMAP to 5 dims (preserves local structure, dramatically speeds HDBSCAN)
• HDBSCAN on 5-dim representation
• c-TF-IDF: TF-IDF where 'document' = all text in a cluster Produces top-N discriminative words per cluster as topic representation

Optional LLM topic naming: Pass top-10 c-TF-IDF words to an LLM: 'These words describe a topic: refund, payment, charge, billing, invoice, dispute. Name this topic in 3-5 words.' → 'Payment and Billing Disputes'

Hyperparameter guide:

• UMAP n_components: 5 for clustering, 2 for visualization
• UMAP n_neighbors: 10-50 (higher = more global structure)
• HDBSCAN min_cluster_size: ~0.5-1% of corpus size (start here)
• If noise > 40%: lower min_cluster_size
• If too many micro-clusters: raise min_cluster_size