Who this is for
- Teams that shipped a GPT feature and now can't tell if it's actually working.
- Founders looking at their OpenAI bill and trying to work out why it's 4x last month.
- Support leads getting tickets they can't reproduce because the model said something different yesterday.
- Companies building diagnostic / assessment / advisory tools on top of LLMs where wrong answers have real consequences.
What I build
01 — Eval harness from day one
A fixed set of test inputs that runs against every prompt change, model change, retrieval change. Scores include correctness, refusal rate, latency, token cost, and any domain-specific metric that matters (factual grounding, format compliance, safety).
Without this, you're flying blind. Every prompt tweak might be making things better in one place and worse in three others. You won't know until users tell you.
02 — RAG that actually retrieves the right thing
Chunking that respects document structure. Hybrid retrieval (vector + BM25 + metadata filters). Reranking. Source citations the user can click. And — most underrated — an admin view of "what did we retrieve for this user query?" so you can debug bad answers in seconds, not hours.
03 — Hallucination containment
A short list of patterns that catch the worst classes of made-up answers before they ship:
- Grounded refusals — model must cite a retrieved chunk or refuse.
- Structured outputs with schema validation, so format failures fail loudly instead of silently lying.
- Self-consistency checks for high-stakes answers.
- Human-in-the-loop escape hatches for the long tail.
04 — Cost & latency guardrails
Per-user budgets. Per-request token caps. Semantic caching for the questions everyone asks. Model routing — cheap model handles the 80%, expensive model handles the long tail. Circuit breakers that fall back to cached answers or a cheaper model under spike load instead of taking down the feature.
05 — Prompt versioning, like real code
Prompts in version control. Diffs reviewable. Rollouts gated by eval scores. A/B tests where the winner is decided by metrics, not vibes. Treat prompts like the code they actually are.
06 — Observability your team will actually use
Per-conversation traces. Token cost per user. Slowest 1% of requests. Eval regression alerts. A flag-this-answer button that creates a reproducible test case. The boring stuff that makes the loud stuff stop being scary.
How I usually engage
- Week 1 — Diagnostic. Read the code. Run the prompts. Look at the bill. Tag the worst 20 cases from support.
- Week 2 — Eval harness + cost ceiling. The two things that should exist before anything else.
- Week 3+ — Targeted fixes, ordered by user pain and ROI. RAG quality, hallucination containment, caching, model routing, whichever is bleeding.
- Handover. Documented prompts, documented evals, runbooks. Your team owns it after.
What I bring to it
Background building production diagnostic and assessment systems on top of GPT-class models (Oracle of Founders is one of mine, plus several private client systems). Backed by The Gyld, my Athens studio, when execution scope grows.
What I won't do
- "Just plug GPT in" projects where there is no eval, no metric, and no plan for what happens when it answers wrong.
- Demos for fundraising decks. Build a real feature or don't.
- Agent frameworks for the sake of agent frameworks. If a single well-evaluated prompt does the job, that's the job.
Frequently asked
Do you only work with OpenAI?
No — OpenAI, Anthropic Claude, open models via vLLM / Ollama, and hybrid setups where a cheap model handles 80% of traffic and an expensive one handles the long tail.
Can you cap LLM costs?
Yes. Per-user budgets, per-request token caps, semantic caching, model routing, and circuit breakers under spike load.
How do you measure hallucination in production?
An eval harness that runs a fixed test set against every change, plus a sampled-from-prod pipeline that re-scores live traffic. You see regressions before users do.
Will you replace my prompt engineer?
No — I work alongside them. My job is the surrounding infrastructure so their prompt work compounds instead of getting overwritten.
Want your LLM feature to stop scaring you?
Send a short note: what it does, what stack it's on, and what's currently going wrong.
Email Eirini → Back to main site