When a company decides to build its own AI, it usually starts with the model: it connects an API, names the assistant and puts it in a chat. Without the other layers, a month later the assistant confidently quotes policies that do not exist, tells a new hire what their manager earns and recommends a competitor to a customer.
Who we modelled.
- A B2C product with 150 people and one assistant for support, staff and analytics.
- 2,000 requests a day (about 48M tokens), growing 50% a year.
- 4 model calls per request, about 24k tokens.
- About 5,000 documents. SSO, a data warehouse with dbt, a helpdesk and a cloud are already in place.
- A small model via API at $1 / $5 per million tokens, or your own open-weight model on 2–4 GPUs.
- An engineer costs $600 a day.
Every figure below is an order of magnitude. How we got them is in the footnote at the end.
Fifteen layers and what breaks without each
Hover over a layer or tap it
Context
Knowledge base
What it does
documents, the data catalogue and the links between them; finds what the user may see
Without it
makes up policies and prices
Done when
right source in the top 5 for 90% of test questions
- build
- $9–16k
- upkeep a month
- $1.8–3k
- infrastructure a month
- $150–300
- startup
- $184k
- marketplace
- $574k
- Open source
- OpenSearch, pgvector, DataHub, Neo4j
- SaaS
- Pinecone, Atlan, Azure AI Search
All fifteen: $55–104k to build, $10–20k a month of upkeep and $1.9–5.4k a month of infrastructure.
Most of the money goes to context (46%) and the agent (28%). The model via API costs almost nothing to build; its cost shows up later in the token bill.


Answers in text and answers in numbers
Different layers handle these questions:
- "what does the policy say": the knowledge base, which searches documents the user is allowed to see;
- "how many orders did we have in October": the semantic layer, which holds agreed metric definitions.
This is where most teams trip. If the agent looks for numbers in text or writes its own SQL, its answers sound right and are wrong, and two colleagues get different figures for the same question.
Three rollout stages
If you lack the resources for the whole stack at once, you can still start. The layers go in over three stages, and after each stage the system is already in use.


Pilot, $16–30k
A document assistant for one team. The model, the prompt, evals and traces go in fully. Lite versions: a knowledge base for one team with permissions taken from the source, a harness with one call and no loop, 2–3 tools and skills in git, an internal chat, basic injection rules, and a gateway with keys and a log of every call. Personal data is not masked yet in the pilot, so it runs on internal documents with no customer data. Done when the team uses the assistant every day and the reference questions pass.
Production, another $22–40k
An assistant in the support chat that takes actions, remembers the customer, respects permissions and passes hard cases to an agent. The pilot's lite layers become full ones: the knowledge base gets owners and freshness checks, tools and skills get a catalogue with access and versions, and the filter blocks injections. The gateway masks personal data before it reaches the model, holds the budgets and switches to a fallback model when a provider fails.
Scale, another $17–34k
Answers in numbers from the warehouse, and control of the token bill. The gateway gets a cache and routing once tokens cost more than about $3k a month. Feedback turns bad answers from users into tests. Done when the agent's numbers match the reports and user ratings reach the evals every week.
Four rules we keep:
- every stage goes live with real users;
- evals and traces go in before the first prompt change;
- the next stage starts only after the "done when" condition holds;
- numbers come last: without a semantic layer the agent guesses the SQL.
Open source, hybrid or SaaS
The stack can be assembled three ways:
Open source
Everything is yours, including an open-weight model on your own GPUs (Qwen, Kimi, DeepSeek, GLM).
Hybrid
Your own open-weight model takes the simple tasks, the hard ones go to an external API, and chat, filters and observability are bought as SaaS.
SaaS
Everything sits with vendors.


For the startup at 2,000 requests a day:
| Open source | Hybrid | SaaS | |
|---|---|---|---|
| Build | $60–115k | $50–96k | $34–64k |
| Over 5 years | $1.61–2.35M | $1.26–2.17M | $1.13–2.62M |
| Where the money goes | work 60%, infrastructure 40% | work 52%, licences 23%, infrastructure 15%, tokens 9% | licences 51%, work 31%, tokens 17% |
| Pros | data stays inside, no token bill | half the tokens do not depend on vendor prices, personal data stays inside | cheapest first year, fewest people |
| Cons | most expensive at this volume, GPUs and MLOps are on you | three things to operate, part of the bill grows with traffic | the bill grows with traffic, 6–10 vendors and contracts |
| Choose when | data cannot leave, or the API costs more than $10–20k a month | personal data must stay inside and volume is below the GPU break-even | you need to test an idea fast |
At the central estimate the hybrid is the cheapest at $1.69M over five years, SaaS costs $1.83M and open source $1.97M. They land within about 15% of each other, so at this volume you choose the stack by your team and your data rules. SaaS has the longest whisker: its bill is tied to requests and seats and grows fastest by year five.
We ran the same stack for two more companies: a mid-size one (1,000 people, 10,000 requests a day) and a marketplace (3,000 people, 50,000 requests a day, about 2B tokens, growing 30% a year, 5 calls of 8k tokens, about 200,000 documents, three engineers on the platform team). Five-year cost:
| Company | Requests a day | Open source | Hybrid | SaaS | Best fit |
|---|---|---|---|---|---|
| Startup, 150 people | 2,000 | $1.97M | $1.69M | $1.83M | hybrid |
| Mid-size, 1,000 people | 10,000 | $2.4M | $3.0M | $4.1M | open source |
| Marketplace, 3,000 people | 50,000 | $4.7M | $9.8M | $16.2M | open source |
The mid-size company is interpolated between the startup and the marketplace by token volume. The marketplace is seven times above the break-even point for your own model: tokens become the biggest cost, and open source costs less than a third of SaaS.
The layers are the same in all three cases; request volume decides which option is cheapest.
The hybrid's cost layer by layer, for the startup and the marketplace over five years, is in the pyramid above: hover over a layer.
Token cost risks
The price per token falls every quarter. Agent loops, memory and reasoning use more tokens per request over time, so the total bill grows.
- Moving to a frontier model triples the bill in one release.
- If token prices double, that adds $316k over five years in SaaS and $160k in the hybrid. Open source does not depend on that price.
- A per-request token budget in the harness, a cost alert in traces, a retention limit in memory and moving some requests to your own model all reduce the risk.
When your own model costs less than the API
The cost of your own model barely changes with volume: two GPUs for redundancy and an engineer. The API bill grows with requests.
For mid-tier models the lines cross at about 7,000 requests a day, about 170M tokens. For frontier-tier models the threshold is about 11,000 requests (about 260M tokens), since a large open-weight model needs 8-GPU nodes.


Your own model
pays off if:
- requests are above that threshold;
- the load is steady, so GPUs do not sit idle at night;
- you have at least one engineer for MLOps;
- data cannot go to an API provider.
The API
stays cheaper if:
- you have fewer than 5,000 requests a day (about 120M tokens);
- the load comes in peaks and GPUs sit idle half the time;
- you need the best model on the market and it has no open weights.
Five takeaways
Start with a pilot for one team
It costs $16–30k: four layers in full and six as lite versions. All fifteen layers cost $55–104k and take three stages.
Evals and traces go in on day one
Without them every prompt or model change is a guess, and breakage reaches customers before you see it.
Context and the agent are ~75% of the build
The work goes into the knowledge base, the semantic layer, the harness and the channel. The model is a small line in the build budget.
Small scale: the stacks cost about the same
At 2,000 requests a day, hybrid, open source and SaaS land within about 15% of each other over five years. Choose by team and data rules.
Compare stacks by cost per resolved request
Same quality bar, all costs in: escalations, repeat contacts, upkeep. Token price alone misleads.
Where to start
The pilot costs $16–30k, roughly one to two months of work at $600 a day:
- a model via API and the prompt in git;
- 20–50 reference questions in evals;
- a log of every step in traces;
- lite versions of the knowledge base, the harness, 2–3 tools and skills, an internal chat, the filter and the gateway.
Add the next layers as problems appear:
- the assistant makes up policies: improve the knowledge base;
- people expect it to take actions: add the tools and skills catalogue and permissions;
- the bill grows: add the cache and the router.
The model will change a couple of times along the way, and the layers stay the same.
You can check which layers you already have in ten minutes: the AI readiness check shows the target for a company your size and the gap to it.
