Module 3 · Inference engineering · Chapter 20 · Free
Why run models yourself? Inference engineering for FDEs
Two ways to get tokens
Up to now this course treated the model as a service you call. A managed API (a hosted model from a provider) means someone else owns the hardware, the software that runs the model and the pager. You send text, you get text, you pay per token.
The alternative is self-hosting an open-weight model. "Open-weight" means the trained model file (its weights) is published so you can download and run it. You rent or own GPUs, run the model on them, and expose your own endpoint. Running a model to answer requests is called inference, and doing it well on purpose is inference engineering. Note that open-weight is not always the same as open source: the license still sets rules, which Chapter 22 covers.
Neither is better in general. They are two different bundles of responsibility. This chapter helps you see which bundle fits a given customer, and why the answer often changes as the customer grows.
Why customers ask for it
- Data residency and isolation. Some data must stay in a region, a customer's own cloud account (a VPC, a private network) or even a network with no internet at all (air-gapped). A model running inside that boundary never sends text out.
- Cost at scale. Per-token pricing is simple and fair at low volume. At high, steady volume, a fixed GPU bill can be cheaper per token. Only if the GPUs stay busy, as the calculator below shows.
- Latency control. On your own deployment you choose where the model runs, how requests queue and which speed tricks to apply. You are not sharing capacity with strangers, so there are fewer surprises.
- Customization. Fine-tuned weights (Chapter 16), a small model specialised for one task, or unusual model types can usually only be served by you.
- Vendor risk. A provider can change prices, retire a model version, throttle you or have an outage. Owning the weights means a model you depend on does not vanish. You trade that for owning the outages yourself.
What you take on
Every reason above has a matching bill in engineering time. If you self-host you now own:
- GPUs. Getting them, paying for them while idle, and fitting the model into their memory (Chapter 25).
- Operations. Deploys, rollbacks, autoscaling, upgrades of drivers and serving software, and cold starts when a new copy has to load a large model.
- Quality. Nobody tests the model on your task for you. You need an eval set (Chapter 18) and you re-run it every time you change something.
- On-call. When the endpoint is slow at 3 a.m., it is your team that looks at it.
The honest summary: a managed API sells you a result. Self-hosting sells you control, and control has to be staffed.
Decide with the customer
You will rarely be asked to pick for yourself; you will be asked to help a customer pick. Walk this tree with their real answers. The "why not the others" line under each result is the part to say out loud in the meeting.
The three layers
"Run the model" hides a lot. It helps to split inference engineering into three layers, from the metal up to the workflow. Step through the stack.
- Real users send prompts of varying length, at varying times. Everything below exists to serve this traffic at the speed and price the product needs.
- The runtime is the part that actually runs the model on a GPU: the model weights, the serving engine and the speed techniques inside it. It decides how fast one copy of the model answers and how many requests it handles at once. Most of Module 6 lives here.
- One fast copy is not a service. Infrastructure adds many copies, load balancing, scaling up and down with demand, multiple regions and a supply of GPUs. It is what keeps latency steady when traffic doubles.
- The layer people feel day to day: how you deploy a new model version, watch latency and errors, run evals before shipping, and see what each request cost.
- A fast runtime on thin infrastructure still gives users slow tails. Great infrastructure with no tooling means nobody notices a quality regression. When you scope a self-host project, ask who owns each layer.
Why scale and specialization change the answer
Two forces push customers from API to self-hosted.
Scale. A managed API price includes the provider's margin and the cost of idle capacity they hold for everyone. When your volume is large and predictable, you can keep your own GPUs busy and that bundle gets less attractive. It also becomes worth spending engineer-weeks on speed-ups, because a 20% gain on a big bill pays for the work.
Specialization. A general frontier model is built to be good at everything. Many products need one narrow job done well: classify tickets, extract fields, answer from a product manual. A smaller model tuned to that job (Chapters 16 and 22) can match a large one on the task, answer faster and cost far less to run. Smaller specialised models are cheap to self-host and often are not offered as a managed product at all.
Both forces apply to the insurer: steady claim volume and one narrow, well-defined reading task.
Workload shape matters
- Online vs offline. Online means a person waits (chat, search). Latency decides. Offline means a batch job (summarize last year's files overnight). Only total throughput and cost per token matter, and you can run GPUs at full tilt.
- Consumer vs B2B. Consumer traffic is large, spiky and follows daily rhythms. B2B traffic is often smaller, steadier, and tied to contracts about data handling and uptime.
- AI-native apps. When the model is the product, inference quality, speed and cost are the business. These teams usually invest in inference engineering early. When AI is one feature of an existing product, a managed API is often enough for longer.
Ask for three numbers early: requests per day, typical prompt and answer length, and how much of the day is busy. Chapter 21 shows how to measure these properly.
Calculator: when does it break even?
Enter your own numbers. This page contains no provider prices, on purpose: use the customer's real quote and your real GPU bill. The values pre-filled are placeholders, not market data. Everything runs in your browser and nothing is sent anywhere.
Model: self-host cost = GPU cost + engineering cost, treated as fixed. API cost = volume x price. Break-even volume = fixed cost / price. Real life adds idle time, scaling steps, discounts and quality differences, so treat the result as a first filter, not a quote.
Recap
- Managed API: someone else runs it, you pay per token. Self-hosted open-weight: you control and you operate.
- Reasons to self-host: data boundaries, cost at steady scale, latency control, custom weights, lower vendor risk.
- What you take on: GPUs, operations, evals and on-call. Control has to be staffed.
- Three layers: runtime (the model on a GPU), infrastructure (many copies, scaling, regions), tooling (deploy, monitor, evaluate).
- Scale and specialization move the answer. Workload shape (online or offline, spiky or steady) decides how busy GPUs stay.
- As an FDE, bring numbers: volume, prompt and answer lengths, data rules, team skills. Offer a hybrid when the evidence is mixed.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.