Evals interview questions
Measuring LLM quality: test sets, graders, regressions. Ten practice questions with answers, for AI Engineer interviews.
Regression test (for prompts)
Show answer
Re-running a fixed eval set after a change to make sure previously good cases didn't get worse.
Run it before every prompt or model change.
What's the main purpose of an eval set for an LLM feature?
- To measure quality consistently so you can compare prompt/model changes and catch regressions
- To train the model
- To reduce API cost
- To replace user testing entirely
Show answer
To measure quality consistently so you can compare prompt/model changes and catch regressions
Without evals, every change is a guess.
Where should test cases for an eval come from?
Show answer
Real or realistic user inputs (from logs, with privacy care), known failure cases and bug reports, edge cases you can anticipate (empty, very long, adversarial, other languages), and a balanced mix of easy and hard cases. Each case needs a clear success criterion.
Evals should look like production traffic plus its sharp edges.
Pass@k
Show answer
The probability that at least one of k sampled attempts succeeds, often used for code generation.
Distinct from requiring every attempt to succeed.
Golden dataset
Show answer
A curated set of inputs with trusted expected outputs or labels used as the reference for evaluation.
Quality of evals depends on quality of this set.
What is 'LLM-as-a-judge'?
- Using a model to grade outputs against a rubric or reference
- Letting users vote on outputs
- A legal review of AI output
- Training a classifier from scratch
Show answer
Using a model to grade outputs against a rubric or reference
Scales grading, but should be validated against human judgments.
Which grading method is best for a classification task with fixed labels?
- Exact-match code-based grading
- LLM-as-a-judge with a long rubric
- Human review of every output
- BLEU score
Show answer
Exact-match code-based grading
Use the cheapest reliable grader; code checks are fast and deterministic.
How do you check that an LLM-as-a-judge grader is trustworthy?
Show answer
Have humans label a sample of outputs, run the judge on the same sample, and measure agreement (e.g. accuracy or correlation). Inspect disagreements to refine the rubric, test for biases like length and position, and re-check agreement when you change the judge prompt or model.
The judge is itself a system that needs evaluating.
Describe how you'd decide whether to switch a feature to a cheaper model.
Show answer
Run the existing eval set on both models with the same prompts, compare quality metrics (and look at failures, not just averages), along with cost and latency. Possibly adjust the prompt for the cheaper model and re-run. Switch only if quality stays within an agreed threshold, then roll out gradually with monitoring and a quick way to revert.
Exactly the process this app uses before downgrading models.
Your LLM judge tends to prefer longer answers regardless of quality. What is this called and how do you mitigate it?
- Verbosity bias; use a specific rubric, calibrate against human labels, and control for length
- Overfitting; add more data
- Hallucination; lower temperature
- Position bias; shuffle the prompt
Show answer
Verbosity bias; use a specific rubric, calibrate against human labels, and control for length
Judges have biases (length, position, self-preference) that you must measure.
Practise these with spaced repetition and AI grading, then try a mock interview.
Start Learning Free