EuclidPRO.ai · Evaluations
Evaluations: how models earn their place
An AI capability ships only after it is measured on the kind of data and hardware it will meet. These are the studies behind EuclidPRO.ai's AI assistants: what was tested, how, and what it changed.
Method
Principles every evaluation follows
Same harness for every model
One prompt, one JSON schema, temperature 0 and a fixed seed. Only the model changes.
Held-out test set
100 labelled GCC industrial-supply emails across 8 categories. Training data is written without ever opening it.
Deterministic scoring
No model grades another on the triage tests: values are normalised (dates, units, currencies) and compared field by field.
Three kinds of hallucination
A value where there should be none, a value not in the email at all, and a real value from the email in the wrong field are counted separately.
Speed measured honestly
Each prompt carries a unique tag so no request reuses a cached prompt; memory and CPU noise are recorded per request.
Decisions written down
Each study states its question, metric, result and the product decision it led to.
Catalogue
Completed and planned evaluations
- Completed2026-10-28
Hardware evaluation: small models on one CPU server
Which small open model can serve triage on a single CPU server, and how many requests can it take at once?
18 models, 444 timed requests at 1 to 5 parallel slots on a 12-vCPU server with no GPU.
- models
- 18
- fastest single-request (qwen3:0.6b)
- 13.9/min
- best throughput, 5 parallel
- 31.9/min
- Completed2026-10-30
Fine-tuned vs base model on the held-out test set
Can a 0.6B model, fine-tuned on CPU, triage real-format business emails reliably?
The same harness, prompt and settings for both models on 100 test emails the training data never saw.
- category accuracy
- 30% → 88%
- schema-valid JSON
- 57% → 100%
- Completed
Content-generation model selection
Which model should generate course, slide and content output, for its quality and its cost?
5 candidate models considered. Claude Opus 5.5 judged the quality of sample generated data, and token use and price were compared.
- chosen model
- Haiku 5.5
- judge
- Opus 5.5
- Planned
Accuracy benchmark: 18 models on 100 emails
Which off-the-shelf small model triages best, at what speed, before any fine-tuning?
Harness, dataset and scoring are built and validated; the full run publishes to doculense.ai.
- Planned
GPU serving: the same models on an RTX 5090
What does a single GPU change in latency, parallel throughput and cost per email?
Same emails and scoring as the CPU evaluation, served with llama.cpp and vLLM.
- Planned
Thinking mode on and off
Does step-by-step reasoning improve triage enough to pay for its extra tokens?
Paired runs of reasoning-capable models with thinking enabled and disabled.
- Planned
Adversarial robustness
How do models handle prompt injection, spoofed senders and misleading subjects?
Breakdown of the test set by trap type, for the base and fine-tuned models.
Study · Model selection
Choosing the content-generation model
The training-content pipeline generates in three stages (course and module plan, slide blueprint, slide content). Each candidate generated sample data, Claude Opus 5.5 judged its quality, and token use and price were compared.
| Model | Outcome |
|---|---|
| Claude Haiku 5.5 | Chosen: the best balance of output quality and cost, although it uses about 30% more tokens |
| ChatGPT Luna 6 | Priced like Haiku 5.5; Haiku gave better results for this use case |
| Gemini 3.8 Flash | Dropped: too expensive |
| Gemini 3.5 Flash Lite | Dropped: too expensive |
| DeepSeek Flash | Considered |
Decision
Claude Haiku 5.5 generates the content. It uses about 30% more tokens, but it gave the best balance of output quality and cost. ChatGPT Luna 6 is similarly priced, and Haiku gave better results for this use case. Gemini 3.8 Flash and Gemini 3.5 Flash Lite were dropped as too expensive.
Judge
Claude Opus 5.5 assessed the quality of sample data generated by each candidate. An LLM judge has biases of its own, so its verdict was weighed together with token use and price rather than on its own.
Public benchmark
The triage benchmark is published live
The email-triage harness behind these studies runs in the open on doculense.ai: dataset validation, per-email results, latency and the per-request hardware data.
doculense.ai