System design for AI apps interview questions
Putting it together: architectures for production LLM systems. Ten practice questions with answers, for AI Engineer interviews.
Rate limiting
Show answer
Capping how many requests a user or client can make in a time window, to protect cost, capacity, and fairness.
Every AI endpoint needs it.
In a production LLM app, where should prompts and model IDs ideally live?
- In versioned config/code so changes are reviewed and evaluated
- Hard-coded in many places
- In the browser
- Only in a teammate's notes
Show answer
In versioned config/code so changes are reviewed and evaluated
Prompts are code: version them and tie changes to eval runs.
Sketch the components of a simple customer-support chatbot backed by a help-centre knowledge base.
Show answer
A chat frontend with streaming; a backend API with auth and rate limiting; an ingestion job that chunks and embeds help articles into a vector store with metadata; a retrieval step per question; an LLM call with a system prompt, retrieved context, and conversation history; logging and feedback capture; an eval set; and escalation to a human agent when confidence is low or the user asks.
Cover data, serving, quality, and fallback paths.
Async job queue (for AI work)
Show answer
Putting slow AI tasks into a background queue processed by workers, returning results later via polling, webhooks, or notifications.
Keeps web requests fast and handles retries.
Idempotency key
Show answer
A unique key sent with a request so retries don't perform the same side effect (like a charge or a message) twice.
Important for webhooks and payments.
A document Q&A product has 10,000 customers, each with private documents. What's essential in the retrieval layer?
- Tenant isolation: filter retrieval by customer/permissions on every query
- A single shared index with no filters
- Storing documents in the prompt
- Using only the largest model
Show answer
Tenant isolation: filter retrieval by customer/permissions on every query
Data leaks between tenants are the worst-case failure.
Your LLM provider has an outage. Which design keeps your product working best?
- Timeouts, retries with backoff, and a fallback model or degraded mode
- Retry forever
- Show a blank page
- Cache nothing
Show answer
Timeouts, retries with backoff, and a fallback model or degraded mode
Plan for failure of an external dependency.
How would you roll out a new prompt version safely?
Show answer
Version the prompt, run the eval suite against the current version and check for regressions, then release behind a feature flag to a small percentage of traffic, monitor quality signals (feedback, error and refusal rates, cost, latency), compare against the old version, and ramp up or roll back quickly.
Treat prompt changes like code deploys.
Design the backend for an AI mock-interview product that must handle long conversations cheaply and reliably.
Show answer
Store each turn append-only in a database and resend the exact history on each request; cache the stable system prompt and conversation prefix with prompt caching; stream responses to the client; cap the number of turns and output length; enforce auth, per-user rate limits, and plan quotas atomically with refunds on failure; grade the transcript at the end with a structured-output call at higher effort; log tokens and cost per session; and validate prompts and models with a fixed eval set before changes.
This is how this app is built.
Users get charged usage even when the AI call fails. What's the cleanest fix?
- Reserve usage atomically before the call and refund it on failure
- Never count usage
- Count usage only on the client
- Ask users to report failures
Show answer
Reserve usage atomically before the call and refund it on failure
Atomic reservation prevents races; refunds keep billing fair.
Practise these with spaced repetition and AI grading, then try a mock interview.
Start Learning Free