LLM fundamentals interview questions
Tokens, context windows, sampling, and how models generate text. Ten practice questions with answers, for AI Engineer interviews.
What is a token, in the context of an LLM?
- A unit of text the model reads and writes, often a word piece
- Exactly one word
- One character
- An API key used for billing
Show answer
A unit of text the model reads and writes, often a word piece
Tokenizers split text into sub-word pieces. Common words may be one token; rare words split into several. Billing and context limits are counted in tokens, not words.
What does raising the sampling temperature do?
- Makes output more random by flattening the probability distribution
- Makes the model more accurate
- Increases the context window
- Makes the model respond faster
Show answer
Makes output more random by flattening the probability distribution
Temperature divides the logits before softmax. Higher values flatten the distribution so less likely tokens get picked more often; it doesn't add knowledge or speed.
Why do LLM APIs usually charge differently for input tokens and output tokens?
Show answer
Input tokens can be processed in parallel in one forward pass, while output tokens are generated one at a time sequentially, each needing its own forward pass. Output is therefore more compute- and latency-expensive per token, so it's priced higher.
Understanding this helps you design prompts that are long-input, short-output when cost matters.
Context window
Show answer
The maximum number of tokens (input plus output) a model can attend to in one request.
Everything the model 'knows' about this conversation must fit inside it.
Explain what a hallucination is and give two practical ways to reduce it in a product.
Show answer
A hallucination is fluent output that is factually wrong or unsupported by the provided context. Ways to reduce it: ground answers in retrieved sources (RAG) and ask the model to cite them; allow and encourage 'I don't know'; constrain output with structured formats; lower temperature for factual tasks; and verify claims with a second check or eval.
Interviewers want concrete mitigations, not 'use a better model'.
Top-p (nucleus) sampling
Show answer
Sampling only from the smallest set of tokens whose cumulative probability reaches p, then renormalising.
It trims the long tail of unlikely tokens while adapting to how confident the model is.
A conversation exceeds the model's context window. What happens if you send it unchanged?
- The API rejects the request or you must truncate/summarise first
- The model silently remembers the overflow in long-term memory
- The model compresses it automatically with no loss
- It works but costs nothing extra
Show answer
The API rejects the request or you must truncate/summarise first
The context window is a hard limit on input plus output tokens. Your application has to decide what to drop, summarise, or retrieve.
Why might the same prompt at temperature 0 still give slightly different outputs across calls?
- Numerical non-determinism in batched GPU inference and backend changes
- Temperature 0 means maximum randomness
- The model learns from each call
- Tokens are assigned randomly per request
Show answer
Numerical non-determinism in batched GPU inference and backend changes
Floating-point order of operations varies with batching and hardware, so near-tied tokens can flip. Models don't learn from individual API calls.
Autoregressive generation
Show answer
The model generates one token at a time, each conditioned on all previous tokens including ones it just generated.
This is why output length drives latency and why early mistakes can compound.
Why does the tokenizer matter when estimating cost and latency for non-English text or code?
Show answer
Tokenizers are trained mostly on certain data, so other languages, code, and unusual strings often split into many more tokens per word or character. That increases cost, latency, and context usage for the same visible amount of text, so you should measure token counts on real samples rather than estimating from word counts.
A 1,000-word document can be very different token counts depending on language and content.
Practise these with spaced repetition and AI grading, then try a mock interview.
Start Learning Free