Evaluation · completed 2026-10-31
Specialist pipeline vs fine-tuned LLM: GLiNER2 and NuExtract
A 205M-parameter GLiNER2 model inside a code pipeline, two NuExtract extractors and a router in front of the fine-tuned 0.6B LLM, scored on the same 100 held-out emails. GLiNER2 classifies best and fastest; the fine-tuned LLM still extracts fields best.
Problem
The fine-tuned LLM triages well, but every email costs about 3 s of CPU and 8 GB of memory. An expert review suggested a small extraction model inside a code pipeline instead.
Hypothesis
A 205M encoder (GLiNER2) that classifies and extracts in one pass, with validation in code, can match the fine-tuned LLM at a fraction of the cost.
Success metric
The same test emails and deterministic scorer as the LLMs: category accuracy, field F1, values not in the email, latency and memory.
Result
Fine-tuned, GLiNER2 classifies 91% correctly (LLM 88%) in 0.40 s. Its fields reach F1 78.1 against the LLM's 87.8; the best extra extractor lifts that to 82.1 at 6.5 s.
Decision
Split the work: GLiNER2 classifies and routes, the fine-tuned LLM extracts the fields. Together: 91% category accuracy and field F1 87.1.
Setup
The pipeline and the variants tested
1 · Clean
HTML, quoted replies and forwarded history are cut in code; sender name, address and subject are kept apart.
2 · Classify
GLiNER2 picks one of 8 categories with a probability; below 0.6 the email goes to the human review queue.
3 · Extract
A schema per email type: an invoice asks for totals and payment terms, a quotation request for items and deadlines.
4 · Normalise and ground
Dates, amounts, currencies, units and references are parsed in code, and any value not found in the email becomes empty.
5 · Route
Confident results go straight on; uncertain ones go to a person, and their corrections are saved as training data.
| Variant | What it is |
|---|---|
| Base Qwen3-0.6B | The untrained LLM with the triage prompt |
| Fine-tuned Qwen3-0.6B | The LoRA fine-tuned LLM: one call returns category and fields |
| GLiNER2 pipeline, zero-shot | 205M-parameter GLiNER2 classifies and extracts with no training; code cleans, validates and routes |
| GLiNER2, fine-tuned classifier | The same pipeline with a GLiNER2 classifier fine-tuned on the training emails |
| + NuExtract-1.5-tiny | A 0.5B generative extractor fills total, currency and payment terms |
| + NuExtract3 | A 4.3B generative extractor fills dates, totals, delivery, payment terms and line items |
| GLiNER2 router + fine-tuned LLM | GLiNER2 decides the category; the fine-tuned LLM's fields are kept; marketing, spam and other skip the LLM |
| Router, category forced into the LLM | As above, but the LLM is re-run with its JSON schema limited to GLiNER2's category |
Method
- Every variant answers the same 100 held-out test emails and is scored by the same deterministic code as the LLM benchmark.
- All tuning (label wording, rules, epoch choice, which fields each extractor fills) was decided on 200 held-out training emails before any test run.
- GLiNER2 runs in PyTorch on the same CPU threads Ollama uses; NuExtract and the LLMs run in Ollama with temperature 0, a fixed seed and Q4_K_M weights.
- Every value an extractor returns must be found in the email text, or it is set to empty: a pipeline can be wrong, but it cannot invent.
Results
All variants on 100 held-out emails
91%
category accuracy, fine-tuned GLiNER2
fine-tuned LLM 88%
92%
macro F1 over 8 categories
fine-tuned LLM 79%
0.40 s
median time per email
fine-tuned LLM 3.0 s
87.1
field F1, router + LLM
GLiNER2 alone 78.1
Category accuracy
- GLiNER2, fine-tuned classifier91
- + NuExtract-1.5-tiny91
- + NuExtract391
- GLiNER2 router + fine-tuned LLM91
- Router, category forced into the LLM91
- Fine-tuned Qwen3-0.6B88
- GLiNER2 pipeline, zero-shot73
- Base Qwen3-0.6B30
Field F1
- Fine-tuned Qwen3-0.6B87.8
- Router, category forced into the LLM87.3
- GLiNER2 router + fine-tuned LLM87.1
- + NuExtract382.1
- + NuExtract-1.5-tiny78.4
- GLiNER2, fine-tuned classifier78.1
- GLiNER2 pipeline, zero-shot75.5
- Base Qwen3-0.6B61.9
| Variant | Category | Macro F1 | Field F1 | Line items | Not in email | Wrong field | Median | Mean | RAM |
|---|---|---|---|---|---|---|---|---|---|
| Base Qwen3-0.6B | 30.0% | 12.9% | 61.9% | 67.0% | 2.5% | 16.7% | 4.0 s | 4.1 s | 8.8 GiB |
| Fine-tuned Qwen3-0.6B | 88.0% | 78.6% | 87.8% | 83.9% | 1.4% | 7.4% | 3.0 s | 3.0 s | 8.3 GiB |
| GLiNER2 pipeline, zero-shot | 73.0% | 69.7% | 75.5% | 51.7% | 0.0% | 10.0% | 0.52 s | 0.61 s | 1.4 GiB |
| GLiNER2, fine-tuned classifier | 91.0% | 91.7% | 78.1% | 56.3% | 0.0% | 10.2% | 0.40 s | 0.48 s | 2.3 GiB |
| + NuExtract-1.5-tiny | 91.0% | 91.7% | 78.4% | 56.3% | 0.1% | 9.9% | 1.7 s | 1.8 s | – |
| + NuExtract3 | 91.0% | 91.7% | 82.1% | 74.2% | 0.0% | 8.0% | 6.5 s | 13.2 s | – |
| GLiNER2 router + fine-tuned LLM | 91.0% | 91.7% | 87.1% | 83.9% | 1.4% | 7.6% | 3.5 s | 3.2 s | – |
| Router, category forced into the LLM | 91.0% | 91.7% | 87.3% | 82.8% | 1.7% | 7.5% | 3.9 s | 3.5 s | – |
RAM is the peak of the serving process. It is left out for combined variants, where GLiNER2 and an Ollama model run in separate processes. Highlighted: the combination chosen.
Classifier training
Fine-tuning GLiNER2 as the classifier
Full fine-tune of fastino/gliner2-base-v1 on 1,800 training emails for the 8 categories, 4 epochs on CPU in 1.9 hours. 200 training emails were held out to choose the epoch.
Accuracy on held-out training emails, by epoch
| Epoch | Accuracy | Macro F1 |
|---|---|---|
| 1 | 94.0% | 92.1% |
| 2 | 96.0% | 95.2% |
| 3 | 98.0% | 97.3% |
| 4 · chosen | 98.5% | 97.8% |
From practice to exam
- Zero-shot, with only label descriptions, GLiNER2 classified 73% of the test emails correctly. Fine-tuned: 91%.
- On held-out training emails the chosen epoch scored 98.5%; on the test emails 91%. Test emails were written separately, and that gap is the real measure.
- The fine-tuned model is very confident: 87 of 100 test emails passed the 0.6 threshold, so the review queue needs a calibrated threshold before client use.
Findings
What the numbers say
A small encoder classifies as well as the LLM, about 7× faster
Fine-tuned GLiNER2: 91% accuracy and 92% macro F1 in 0.40 s, against 88% and 79% in 3.0 s. It is clearly better on the rare categories: to review F1 96% vs 67%, marketing 80% vs 25%.
Extraction still needs a generative model
Span extraction loses on lists and context: line items 50% correct against 77% for the LLM. NuExtract3 (4.3B) lifts line items to 70% and delivery location to 66%, but takes 6.5 s per email (mean 13.2 s). NuExtract-1.5-tiny added almost nothing.
Grounding removes invented values, not wrong ones
No GLiNER2 or NuExtract variant returned a value missing from the email (0.0%, against 1.4% for the LLM). Real values in the wrong field remain: 10.2% for GLiNER2, 7.4% for the LLM.
Telling the LLM the category does not improve its fields
Forcing GLiNER2's category into the LLM's JSON schema gave field F1 87.3, the same as simply combining the two answers (87.1). The LLM extracts the same values whatever category it is given, so the simple combination wins on cost.
Per category
Category F1 by variant
| Category | Emails | Base Qwen3-0.6B | Fine-tuned Qwen3-0.6B | GLiNER2 pipeline, zero-shot | GLiNER2, fine-tuned classifier |
|---|---|---|---|---|---|
| Quotation request | 28 | 40% | 98% | 91% | 88% |
| Purchase order | 22 | 38% | 98% | 70% | 88% |
| Invoice | 12 | 0% | 96% | 71% | 96% |
| Complaint | 10 | 0% | 95% | 57% | 100% |
| Marketing | 6 | 25% | 25% | 29% | 80% |
| Spam | 6 | 0% | 50% | 100% | 86% |
| Other | 4 | 0% | 100% | 86% | 100% |
| To review | 12 | 0% | 67% | 54% | 96% |
Limits
What this evaluation does not show
- 100 test emails give about ±7 points of uncertainty on accuracy, and several categories have only 4 to 12 emails: 91% against 88% is a tie, the macro F1 gap is the clearer signal.
- The same test set has now scored around ten variants. Every choice was made on held-out training emails first, but before client use the result will be confirmed on a fresh, independently written test set.
- The training and test emails are synthetic, written by AI models to a detailed specification. Real inboxes will be messier.
- Latencies are on one 12-vCPU server with no GPU; GLiNER2 runs in-process, the LLMs through Ollama's HTTP API.