Classifying Documents into 30,000+ Categories (Beyond Prompt Engineering)
Asking an LLM to zero-shot classify a document into one of 30,000+ categories simply fails. We replaced the naive prompt approach with a multi-stage pipeline — vector search, algorithmic guardrails, and tightly-constrained LLM reasoning — plus a human-in-the-loop app, turning a high-cardinality problem into a reliable, continuously-improving system.
The challenge
Everyone loves the idea of throwing a prompt at a model and letting it do the heavy lifting — but you can’t prompt-engineer your way out of a high-cardinality data problem. Stuff tens of thousands of category definitions into a prompt and you blow up the context window, destroy latency, and watch the model hallucinate wildly.
Our approach
Vector-search foundation
Instead of prompting immediately, we vectorized the entire taxonomy and used semantic similarity to narrow tens of thousands of categories down to a small, highly-probable shortlist.
Algorithmic guardrails
Before the LLM ever touched the text, strict similarity thresholds filtered out noise and graded the confidence of potential matches — cheap, deterministic math doing the coarse work.
Chained, constrained LLM reasoning
We broke the LLM’s job into stages: first summarize the complex input, then compare that summary only against the pre-filtered shortlist — forcing the model to output its final prediction in a strict JSON format.
Human-in-the-loop MLOps
We packaged the pipeline into a custom internal web app so domain experts could inspect, validate, and correct the AI’s batch predictions — continuously improving the dataset over time.
The outcome
The result is a reliable, high-cardinality classifier that stays inside context and latency budgets, doesn’t hallucinate categories, and gets better over time as domain experts correct it — an outcome pure prompting could never deliver.
Technology
Generative AI is powerful, but it isn’t a silver bullet. Sometimes the most reliable AI architecture is a rigid mathematical pipeline feeding into a highly restricted model.