Next-generation AI

Custom AI systems that hold up after the demo

We build retrieval, document extraction and agent systems on top of frontier model APIs — together with the eval harness, the guardrails and the tracing that decide whether any of it survives contact with your real documents. We ship an agent only where a workflow provably cannot do the job, because multi-agent orchestration burns roughly 15x the tokens of a chat call and a single agent about 4x — Anthropic’s measured figures. Against a retrieval workflow the gap is smaller, but it is real, and it comes with failure modes a workflow does not have.

Book a call See our projects →

30 minutes with the engineer who would do the work — not a salesperson. No obligation, and you keep whatever we work out on the call.

Engineers reviewing evaluation results before a retrieval system goes to production

What we design against

5.7% → 1.9%: Retrieval failure at top-20: naive embeddings versus contextual embeddings, BM25 and a rerankerAnthropic’s published contextual-retrieval benchmark, measured as 1 − recall@20. Not a Palamed client result — it is the target we design against and then measure on your own corpus.
under 25%: Agent success at pass^8 on tau-bench retail, against over 50% single-pass — GPT-4o, 2024Published tau-bench figures for GPT-4o. Current frontier models score materially higher, and the absolute number matters less than the shape: pass^8 is always far below pass^1, and that gap does not close with model quality. Run-to-run consistency, not peak capability, is what breaks agents in production — so we report pass^k on your tasks, never a one-shot demo number.
0.1x: Input cost on a cached prompt prefix, against 1.25x to write a five-minute cacheAnthropic’s published caching rates. Invalidation cascades tools → system → messages, so one timestamp in the system prompt silently zeroes the hit rate; we instrument cache_read_input_tokens from week one.
95%: Enterprise GenAI pilots that produced no measurable P&L impactMIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" — 52 structured interviews, 153 surveyed leaders, 300+ public deployments. Realised ROI concentrated in back-office process elimination, while over half of budgets went to sales and marketing tools. Not a Palamed result.

What we build

Two analysts reviewing a data table together on a monitor

Retrieval that survives your document estate

Most incidents reported as "the AI hallucinated" are retrieval misses. We parse your estate layout-aware with Docling or Unstructured, tag every chunk with document ID, revision and effective date, then run contextual retrieval: a model writes a 50–100 token preamble per chunk before embedding. That is a one-off contextualisation pass whose cost scales with your small-model input rate — Anthropic measured $1.02 per million document tokens on 2024 Haiku pricing; we recompute it against current rates and your actual corpus size before you approve it, because it is a real line item on a large estate. Retrieval is hybrid — BM25 for part numbers, dense vectors for paraphrase — fused with reciprocal rank fusion, then reranked by a cross-encoder. The reranker is the latency you pay for precision, so it is budgeted rather than discovered: retrieval 50–200ms, reranking 80–300ms over the top ~100 to select 10–20, and time to first token under 700ms end to end, because past roughly 1.5s users assume it is broken.

A labelled question set on your corpus, with recall@20 and context precision measured before and after — recall alone is a trap, because a looser top-k raises it while feeding the generator distractors that raise hallucination at the same time
Effective-date and revision filters in a filterable HNSW index, so superseded SOPs are unreachable rather than merely ranked lower
Answers that carry resolvable chunk citations, and an explicit abstention when no chunk supports them

Engineer annotating a system integration diagram on paper

Document extraction with deterministic validators

Extraction runs against a strict JSON Schema with additionalProperties false, so the schema is the interface contract with your ERP rather than a hope. Deterministic validators run after the model, never instead of it: IBAN mod-97, VAT-number checksum, line items reconciled against the stated total, supplier matched to master data. Confidence comes from validator agreement and cross-field consistency, not from the model self-report, which is unreliable.

A JSON Schema, agreed with whoever owns the ERP fields, that downstream systems can trust
A validator suite that decides what is straight-through and what reaches a human
A review queue showing the source page region, so a person confirms instead of re-keying — and every correction lands in the regression set

Workshop session around printed process diagrams on a table

Agents, only where a workflow cannot reach

We default to a workflow and make the agent justify itself. Where the step sequence genuinely cannot be predetermined, tools stay narrow and idempotent, and privilege separation is enforced by which MCP server exposes them, because MCP’s OAuth model authorises the server rather than the individual tool — with audience-validated tokens and no passthrough of your client’s token to a downstream service. Writes are human-gated in the first release, and the loop runs on durable execution so a process restart resumes from a checkpoint rather than replaying side effects. Termination is explicit — step budget, token budget, no-progress detector.

A tool inventory where each tool is narrow, documented, idempotent, and reachable only through the MCP server that matches its trust boundary
Explicit termination conditions, because MAST attributes 17.1% of multi-agent failures to step repetition and 9.8% to termination unawareness
OpenTelemetry GenAI spans on every run, so a failure is reconstructable instead of anecdotal

Engineer reviewing a monitoring dashboard at a desk at night

The eval harness you keep

The deliverable is not a prompt, it is a regression gate. We read 100+ of your real failed traces and open-code the failure modes — coding converges around 30 traces, when nothing new appears — then turn them into a golden set of 150 to 400 binary pass/fail cases written with your domain expert. Judges are validated on true-positive and true-negative rate against those labels, because raw agreement misleads under class imbalance.

A golden set in your repository, owned by a named person on your side
Cheap deterministic assertions on every commit: format, forbidden terms, citation presence, latency
A CI gate sized so it can actually fire: the blocking threshold is set from the measured variance of the suite, not picked as a round number, and where the golden set is too small to resolve a small delta we say so and gate on the deterministic assertions plus per-failure-mode subsets instead of a blended score

Engineer walking a client through documentation on a laptop

Guardrails, unit cost and the AI Act position

Prompt injection is an architecture problem, not a filtering problem: adaptive attacks broke most of twelve published defences over 90% of the time, and a 500-person human red team broke all twelve. So the agent reading untrusted documents is not the agent holding write access to your ERP — the Rule of Two, enforced in the topology rather than promised in a policy. Cost is engineered the same way: per route, per resolved task.

Adaptive red-team results from promptfoo, mapped to OWASP Top 10 for LLM Applications 2025
Exact model IDs pinned rather than aliased, gateway fallback chains, and champion-challenger evaluation before any version change
An Annex III classification with its reasoning, and Article 50 disclosure built into the templates rather than written into a policy PDF

Systems we work with

We integrate with what you already run. If a platform below is missing, tell us — the pattern usually transfers.

What we have actually run: models and retrieval

Anthropic Claude API and OpenAI structured outputs (strict JSON Schema)
PostgreSQL with pgvector, hybrid BM25 plus dense vectors fused with reciprocal rank fusion
Cohere Rerank for cross-encoder reranking, including Bulgarian
Docling and Unstructured.io for layout-aware document ingestion

What we have actually run: evaluation, tracing and guardrails

Ragas and promptfoo, including adaptive red-team plugins
Langfuse or Arize Phoenix, self-hosted where data residency requires it
OpenTelemetry GenAI semantic conventions
Pydantic and Zod for schema validation and constrained decoding
OWASP Top 10 for LLM Applications 2025, NIST AI RMF 1.0 and AI 600-1, ISO/IEC 42001:2023 as the frameworks findings are mapped to

What we would pick next, and the honest limit of this list

Model Context Protocol (JSON-RPC 2.0), with privilege separation enforced by splitting servers per trust boundaryMCP authorises a server, not a tool
Temporal for durable execution, where an agent loop has to survive process death
Amazon Bedrock, Google Vertex AI or Azure AI Foundry where an EU-region endpoint is a procurement requirement
OVHcloud, Scaleway or Hetzner behind vLLM where open weights have to stay in the EU
If a platform you use is missing, it means we have not shipped on it; ask and we will tell you whether the pattern transfers or whether you want somebody else

How the work runs

01
Trace review
1–2 weeks, fixed fee
What you keep
A counted failure taxonomy, open-coded from 100+ of your real traces or misprocessed documents, plus a baseline recall@20 and groundedness score on a labelled question set written with your expert.
The decision it forces
Whether your problem is retrieval, generation, specification or the corpus itself. The four have completely different fixes, and only one of them is a prompt.
When we stop
If the taxonomy shows the dominant failure is missing, contradictory or superseded source content, we hand you the taxonomy and stop, and we will not propose a build on the back of it. No retrieval architecture recovers a document that was never written.
02
Thin vertical slice
3–5 weeks
What you keep
One end-to-end path in production shape — ingestion, hybrid retrieval, reranking, generation, citations, abstention — behind a feature flag, with the golden set and the CI gate already running against it.
The decision it forces
Whether the measured numbers clear the threshold you set before we started: recall@20, context precision, groundedness, abstention rate, p95 latency, and cost per resolved task.
When we stop
If groundedness stays under 0.85 on the golden set after two tuning cycles, we stop, hand over everything built, and you pay for the weeks worked and nothing further — no hardening phase, no retainer, and we will not propose one. Below 0.85 the system fabricates visibly, weekly, and no amount of prompt work hides it.
03
Hardening
4–8 weeks
What you keep
A guardrail architecture written down against the Rule of Two, adaptive red-team results mapped to OWASP LLM Top 10 2025, pinned model IDs with fallback chains, and an EU AI Act classification with its reasoning.
The decision it forces
What the system is permitted to do without a human in the loop, and what it must never do at all.
When we stop
If the use case classifies as Annex III high-risk and you do not want the provider and deployer obligations attached to it, we redesign so the human stays the decision-maker, or we stop.
04
Handover and run
2 weeks, then a monthly retainer
What you keep
A runbook, the eval suite in your repository, dashboards on your own Langfuse or Phoenix instance, and a written re-index and embedding-migration budget in euros and wall-clock hours.
The decision it forces
Who owns the golden set, what the merge gate blocks on, and which metric triggers a rollback.
When we stop
If nobody on your side is named as owner of the golden set, we do not start the run phase. An unmaintained eval suite decays into a false green light inside a quarter.

A demo proves the model can. Production proves that it does, every day.

What you are probably thinking

We already ran a pilot, it hallucinated, and we stopped.

That is usually a retrieval failure diagnosed as a generation failure. The canonical RAG taxonomy separates "the answer was never retrieved" (failure points 1–3) from "the model had it and got it wrong" (4–7), and the fixes share nothing: chunking, metadata and reranking versus prompt and context construction. Error analysis on your pilot traces settles it in days. Sometimes the honest answer is that the pilot was aimed at a question your documents cannot answer, and we will say that too.

Isn’t this just a wrapper around someone else’s API?

The API call is the commodity, and you are right to price it as one. What decides whether the thing works is the retrieval pipeline, the eval harness, the guardrail topology, the observability and the failure handling. Concretely: MAST attributes 41.8% of multi-agent failures to specification issues, and tau-bench put GPT-4o above 50% single-pass but under 25% at pass^8 in 2024 — current models score higher, and the pass^1-to-pass^k gap is still there. Nothing in the model API fixes either number. That gap is the engagement.

How do we know it will not get worse after you leave?

You do not, unless there is a gate. So the deliverable is a golden set of 150–400 labelled cases written with your expert, deterministic assertions on every commit, judges validated at 0.85+ true-positive and true-negative rate, and a CI check whose blocking threshold comes from the measured variance of the suite rather than a round number — on a 200-case set a one-point move is two cases, well inside judge noise, so where the set cannot resolve a small delta we say so and gate on the deterministic assertions and per-failure-mode subsets instead. The honest caveat: a golden set with no named owner rots. If you cannot name that person, buy the audit and not the build.

Why not wait for SAP or Microsoft to ship this in the suite?

For horizontal use cases you often should, and we will say so before you spend anything. Suite products handle generic document chat and meeting summaries competently, and the MIT study found purchased and vendor-partnered tools succeeded roughly 67% of the time, against about a third of that for builds attempted purely in-house — an argument for buying the horizontal layer, and for not attempting the specific workflow alone. What they will never model is your part-numbering scheme, your revision-control rules, your approval thresholds or your ERP exception logic. Buy the horizontal layer; build only that.

What happens when someone hides instructions inside a document we process?

Filtering is not a defence. Adaptive attacks broke most of twelve published injection defences over 90% of the time, and a 500-person human red team broke all twelve. So it is handled architecturally, by the Rule of Two: no component holds untrusted input, sensitive data access and outbound action at the same time. In practice the agent reading supplier PDFs is not the agent with write access to your ERP, and anything that both reads untrusted content and can act is human-gated. Be clear about what that costs you: the fully autonomous version — read the PDF, post to the ERP, no human — is the version we will not build, because there is no known defence that makes it safe. You get two components and an approval step instead. It is slower and it demos worse. If the business case only works with full autonomy, the honest answer is that the business case does not work yet.

When we are the wrong choice

You want one general-purpose assistant over everything the company has ever written. That project has no acceptance criterion, no golden set and no owner, and it is the shape most of the pilots with no measurable P&L impact take.
The use case is Annex III high-risk — CV screening, task allocation, performance monitoring, creditworthiness — and you want it to decide autonomously. We will build the version where a person decides, or we will not build it.
Volume is under a few hundred documents, tickets or cases a month. Below that the eval suite and the run-rate cost more than the manual work they replace, and we would rather say so on the first call than in the third invoice.

Questions we get asked

How do we choose between fine-tuning and RAG?

RAG for knowledge that changes and must be cited; fine-tuning for format, tone and latency on a stable, high-volume task. Parameter-efficient tuning made compute the cheap part years ago — QLoRA fit a 65B model on one 48GB GPU back in 2023, and open-weight models have only got easier since. The cost is the labelled dataset and the eval suite. Two things people miss: you generally cannot fine-tune the frontier hosted model you are calling today, and distilling its outputs onto a smaller model is a licence question before it is an engineering one, so we check the terms before proposing it. In practice the order is RAG first, then distil the settled high-volume path onto a smaller model once corrections have accumulated into a training set.

Can it actually handle Bulgarian, or only in the demo?

We ship Bulgarian-language systems: the digital dictionary at beron.mon.bg, built for the Ministry of Education and Science and in use across Bulgarian schools — open it and check it. That is retrieval over Bulgarian source text in public, not a demo. It is also our only public Bulgarian language-system reference; the rest of our work is a European car marketplace with 300,000+ listings, an NLP module that classifies and drafts answers to trading questions arriving by email and Instagram — about 85% less manual typing on repeat questions, on the client’s own measurement, with a person approving every reply before it sends — and email-marketing automation that cut a beauty brand’s outreach time 60%. Four projects, that is the complete list. Beyond that it has to be measured rather than assumed. Models benchmarked mainly on English typically run 5–15 points lower on Bulgarian, and a single blended quality score hides that completely. We keep a separate Bulgarian golden set, choose embedding and reranking models on measured Cyrillic performance rather than on marketing, and enforce terminology with a validated glossary instead of a prompt instruction.

What will this cost to run at our volume?

Measured per resolved task, not per token, because a cheap request that needs three retries is not cheap. Typical ranges are €0.01–0.05 per cached retrieval answer and €0.20–1.50 per multi-step agentic case. The levers are prompt caching at 0.1x on reads, the Batch API at 50% for anything not latency-sensitive, and routing classification and extraction to a small model. We instrument the number from week one.

Can our data stay in the EU, and will it be used for training?

Both are solvable and both belong in the contract, not in an assurance. EU-region endpoints on Bedrock, Vertex or Azure with zero-data-retention where the provider offers it, Article 28 processor terms, or fully self-hosted open weights on OVHcloud, Scaleway or Hetzner behind vLLM. Self-hosting is a real option with real economics: continuous batching only pays at sustained occupancy, so we model it against your actual queries per second.

Do we need our own ML engineers to keep this running?

No, but you need one named owner for the golden set. The repository is ordinary Python or TypeScript plus a CI job, and the dashboards are Langfuse or Phoenix. The genuinely ML-specific skill is reading traces and labelling them honestly, which a domain expert does better than an engineer. We train that person during handover and leave the annotation guide behind.

How quickly do we see something running in production?

A thin vertical slice behind a feature flag typically lands 3–5 weeks after the trace review, and it is real: your documents, your retrieval, your citations, your golden set. Full rollout depends almost entirely on the state of the source material. If your documents have no revision metadata and no single source of truth, the first weeks are ingestion and governance, not modelling, and we will tell you before you sign.

Keep reading

How we work: stages, artefacts and kill criteria AI advisory: deciding what is worth building first Case studies: what we have actually shipped

What would it take to trust this in production?

Send us 100 of the traces your pilot got wrong, or 100 of the documents it misread. We read them, open-code the failure modes and tell you whether the problem is retrieval, specification, or the corpus itself. Under a mutual NDA and an Article 28 processor agreement signed before anything is sent, processed in the EU, deleted on request or at the end of the review by default. If you would rather not move the data at all, we run the review in your environment.