Cost, latency and caching interview questions
Token economics, prompt caching, batching, model routing. Ten practice questions with answers, for AI Engineer interviews.
Time to first token (TTFT)
Show answer
How long until the first token of the response arrives; what users perceive as responsiveness in streaming UIs.
Long inputs increase TTFT; caching can reduce it.
Which usually has the biggest effect on response latency?
- The number of output tokens generated
- The length of the API key
- The time of day
- Using JSON instead of text
Show answer
The number of output tokens generated
Output tokens are generated sequentially; shorter outputs are faster.
Estimate the cost of a call with 2,000 input tokens and 500 output tokens at $3 per million input and $15 per million output.
Show answer
Input: 2,000 × $3 / 1,000,000 = $0.006. Output: 500 × $15 / 1,000,000 = $0.0075. Total about $0.0135 per call.
Being able to do this quickly is expected in interviews.
Semantic caching
Show answer
Reusing a stored answer when a new query is similar enough (by embedding) to a previous one, skipping the model call.
Risky when answers depend on user-specific context.
Model routing
Show answer
Sending each request to the cheapest model that can handle it well, e.g. small model for easy cases and large model for hard ones.
Requires a way to judge difficulty and evals per route.
How does prompt caching save money?
- A repeated prompt prefix is processed once and reused at a discounted rate on later calls
- It caches the final answer and skips the model
- It compresses the prompt
- It lowers output token prices
Show answer
A repeated prompt prefix is processed once and reused at a discounted rate on later calls
Put stable content (system prompt, documents) first so the prefix matches exactly.
You put the current timestamp at the top of your system prompt. What happens to prompt caching?
- The prefix changes every request so the cache never hits
- Nothing changes
- Caching gets faster
- The API rejects the request
Show answer
The prefix changes every request so the cache never hits
Caching needs an exact prefix match. Put changing data after the cached part.
Name four ways to reduce LLM cost in a production app.
Show answer
Use prompt caching for repeated prefixes; route easy requests to smaller, cheaper models (validated by evals); cap and reduce output length; use batch APIs for non-urgent work; trim unnecessary context and retrieved chunks; and cache full responses where safe. Track cost per feature and per user.
Any four well-explained ones are fine.
A chat feature feels slow. How do you investigate and improve latency?
Show answer
Measure first: break latency into network, queueing, time to first token, and generation time, and look at input and output token counts. Then stream responses, shorten outputs, cache stable prefixes, reduce context size, lower reasoning effort or use a faster model where quality allows, run independent calls in parallel, and move non-urgent work off the request path.
Measure, then optimise the biggest component.
You need to classify 2 million documents overnight; nobody is waiting. What's the best approach?
- A batch API at a discount, with a small model validated on an eval
- Real-time calls with the largest model
- Streaming
- One request with all 2 million documents
Show answer
A batch API at a discount, with a small model validated on an eval
Batch processing trades latency for lower cost; model routing cuts cost further.
Practise these with spaced repetition and AI grading, then try a mock interview.
Start Learning Free