Python AI architecture & engineering
AI engineering.
Solved in Python.
Build a RAG pipeline. Troubleshoot retrieval. Set up fine-tuning and distillation. Practice the engineering work that makes AI systems useful.
pick a problem. write the fix.
Engineering scenarios
7 scenarios
Repair a RAG chunker that splits answers across chunks
A support assistant stopped finding multi-step runbook procedures after ingest switched to fixed-size character windows. A worked Python example packs already-parsed blocks into chunks under a budget without cutting a procedure: five inline blocks, one short function over a list of dictionaries, its exact output, a line-by-line walkthrough, and one practice variation.
- Greedy packing under a prompt budget
- Records as a list of dictionaries
- A loop with a guard, a join test, and continue
Fuse keyword and vector rankings with reciprocal rank fusion
A support assistant retrieves runbook chunks from a keyword index and a vector index, and taking the vector list first buried exact matches for error codes. A worked Python example fuses two ranked lists with reciprocal rank fusion: two inline lists, a dictionary that accumulates scores, a sort with a key function, its exact output, a trace of every contribution, and one practice variation that weights the lists.
- Reciprocal rank fusion of two retrievers
- A dictionary as a score accumulator with get and a default
- Sorting with a key function and a tuple tie-break
Compare two prompt versions ticket by ticket, not by accuracy alone
A rewritten classification prompt scored higher on the evaluation set and shipped, then misrouted the outage tickets the old prompt got right. A worked Python example pairs the two prompts item by item: nine inline tickets, zip with strict length checking, a four-way count keyed by tuples, the regressions and gains by id, its exact output, and one practice variation that normalizes answers before comparing.
- Paired comparison of two model versions
- zip over parallel lists with strict checking
- Tuples as dictionary keys for a two-by-two count
Retry a flaky model call with exponential backoff and a cap
An assistant retried a hosted model in a tight loop, hammering a rate-limited endpoint and retrying invalid requests forever. A worked Python example wraps one call in a retry loop with exponential backoff: a scripted fake endpoint, functions passed as values, try and except on retryable errors only, a cap that gives up loudly, its exact output, and one practice variation that adds a total time budget.
- Exponential backoff with a retry cap
- Passing functions as arguments (the call and the sleep)
- try, except on chosen exception types, and raising
Split a fine-tuning set by hashing ids so the split never moves
A weekly fine-tuning job reshuffled its train and validation split each run, so examples drifted between sides and validation scores jumped. A worked Python example assigns each record by hashing its id with SHA-256 into a hundred buckets: six inline transcripts, hashlib and integer conversion, a dictionary of lists, proof that new records never move old ones, a seeded shuffle for comparison, its exact output, and one practice variation that adds a test split.
- Stable train and validation assignment by content hash
- hashlib, encode, hexdigest, and int with base sixteen
- Buckets with the modulo operator and a threshold
Validate a JSONL training file line by line and count what you reject
An overnight fine-tuning job crashed on one malformed line of a JSONL export, and after a manual fix it trained on records with empty answers. A worked Python example streams the file with a generator: nine inline lines (eight records and a blank), json.loads inside try and except, checks for missing and empty fields, kept records yielded one at a time, rejections counted by reason with their line numbers, its exact output, and one practice variation that drops duplicate prompts.
- Streaming validation of JSONL records
- json.loads and JSONDecodeError
- A generator function with yield
Calibrate model confidence with a hand-written training loop
An assistant escalates to a human below a confidence of 0.6, but its 0.9 answers were right only seven times in ten. A worked Python example fits a two-parameter calibration to eight scored answers with the loop PyTorch runs for you, forward, loss, gradient, update, written in plain Python: math.exp and the sigmoid, a function that returns two gradients, an epoch loop with running totals, its exact output, a trace of the weights, and one practice variation that stops early when the loss stops improving.
- A training loop by hand: forward, loss, gradient, update
- math.exp and the sigmoid
- Functions that return several values as a tuple
Practicing for an interview? Read, build, test, explain.
Work with the supplied Python files, validate the result, and explain your decisions before revealing the reference solution.
- 01
Understand the system
Read the brief, trace the data, and identify the constraints.
- 02
Work in Python
Build or repair the implementation using the supplied files.
- 03
Prove the result
Run tests, inspect failures, and check the quality of the output.
- 04
Explain the choices
Compare alternatives and defend the engineering tradeoffs.