LLM & Generative AI Interview Questions (With Answers)

AI is the broadest category — systems that perform tasks associated with intelligence. Machine learning is a subset that learns patterns from data instead of hand-written rules. Generative AI is a further subset (typically built on deep learning) focused on generating new content — text, images, audio — rather than just classifying or predicting a number. See our full LLM explainer for the complete breakdown.

A token is the basic unit an LLM processes — often a word fragment rather than a whole word. Tokens matter because context windows, API pricing, and generation speed are all measured in tokens, not characters or words.

Temperature controls randomness in next-token sampling. Low temperature (near 0) makes output deterministic and focused; high temperature increases variety and creativity at the cost of consistency and increased hallucination risk.

Because an LLM predicts the statistically plausible next token, not a verified fact. It has no built-in mechanism to check truth — a fluent, confident-sounding but wrong answer is a normal failure mode, not a bug. Grounding answers with RAG and better prompting are the standard mitigations.

Training is the one-time, expensive process of adjusting a model's parameters on large datasets. Inference is every subsequent use of the already-trained model to generate a response — no learning happens during inference.

Few-shot prompting provides 2–5 examples of the input/output pattern before asking for a new one. Use it when zero-shot prompting produces inconsistent formatting or misses specific edge cases — examples lock in the pattern far more reliably than description alone.

Be explicit about what the model should do when it doesn't know something ("say you don't know" rather than guessing), provide relevant context directly in the prompt, and ask for reasoning or sources where applicable. See the full Prompt Engineering Guide.

Prompt injection is when untrusted input (user text, a retrieved document) contains instructions designed to override the system prompt. Defences include a clear instruction hierarchy, treating retrieved/user content as data rather than commands, and limiting what tools an agent can access.

Ask explicitly for a structured format (usually JSON), show the exact schema or an example, use a native JSON-mode/structured-output API feature where available, and validate the parsed result defensively in code before trusting it downstream.

RAG lets an LLM answer using information outside its training data — private documents, recent information — by retrieving relevant chunks and inserting them into the prompt before generation. It also reduces hallucination since answers can be grounded in retrieved source text.

Chunk documents into manageable pieces, embed each chunk into a vector, store those vectors in a vector database, embed an incoming query the same way, retrieve the most similar chunks, and insert them into the prompt so the LLM generates a grounded answer. Full walkthrough in our RAG Guide.

Separately: retrieval quality (did the right chunks get retrieved — measured with metrics like recall@k) and generation faithfulness (did the answer actually stick to the retrieved context, or hallucinate on top of it). Conflating the two is a common mistake — see our LLM Evaluation Guide.

RAG injects knowledge at query time without changing the model's weights — fast to update, transparent, cheap. Fine-tuning changes the model's weights to shift behaviour, tone, or a specialized skill — better suited to teaching style than injecting frequently-changing facts.

Exact nearest-neighbour search over millions of vectors is too slow for real-time queries. ANN indexes (like HNSW) trade a small amount of accuracy for a large speed improvement, which is why virtually every production vector database uses one by default.

A plain LLM call is prompt-in, text-out — no interaction with the outside world. An agent adds a loop: the model can decide to call a tool, observe the result, and decide what to do next, potentially across multiple steps, before producing a final answer.

You describe available functions (name, description, parameters) to the model. The model can respond by requesting a specific function with arguments instead of plain text; your application code executes the actual function and returns the result to the model to continue reasoning.

Calling the wrong tool, malformed or hallucinated arguments, getting stuck in repetitive loops, and compounding an early small error into a completely wrong final answer several steps later. Bounding step count and validating tool outputs are standard mitigations.

Agentic AI describes systems designed around autonomous, multi-step planning and decision-making toward a goal — as opposed to responding to one request at a time. Frameworks like LangChain/LangGraph provide scaffolding for this, built on top of the same tool-calling and agent-loop mechanics.

A backend service that owns business logic and holds API keys server-side, the LLM provider called only from that backend, and a data layer (database, plus a vector store if using RAG). Clients talk only to your backend, never directly to the LLM provider.

Cache repeated or near-identical requests, track token usage per user/endpoint, set hard spend ceilings, and choose the smallest model that meets quality requirements for each specific task rather than defaulting to the largest available model everywhere.

The input, the exact prompt version used, the model and parameters, the output, latency, and token counts — enough to reconstruct exactly why a given output was produced when behaviour changes or a user reports a problem.

Implement retries with exponential backoff, set sane timeouts, and design a graceful fallback path (a cached answer, a simpler non-LLM response, or a clear "try again" state) rather than letting a slow upstream call hang your whole request.