Recipes, screenplays, law and working with LLMs

Have you ever noticed how when someone follows your favourite recipe, it doesn’t taste quite as good?

A recipe gives you a list of ingredients, quantities, and instructions. It uses prose to record what to do and how to do it. But depending on the ingredients and the chef, a unique interpretation of the dish is produced. Taste, experience, intuition, local ingredients and knowledge will influence the result. Natural language relies entirely on the interpreter’s latent knowledge to fill in the gaps.

Screenplays briefly describe the situation and context: the ambience, the decor, the dialogue, who does what, when, and why. Screenwriters use codified patterns and command words—CUT TO, FADE TO, INT. BATHROOM – MORNING—to structure their documents. The screenplay acts as a reference for the entire production team, from the director and actors to the set designer.

Is it complete and perfect? No. Actors improvise. Directors take liberties. Editing and colour grading change the tone entirely. The screenplay is merely an intermediate artefact; the film is the final interpretation.

Law is an example of a system that has learned to deal with interpretation. Legal texts often begin with “Definitions.” They are organised into Titles, Chapters, Sections, and Clauses that can be referenced and quoted (“see Section 5(b)(ii)”). The legal system figured out something crucial: natural language is not enough. You need an institution downstream to resolve ambiguity, with the authority to settle what the words mean when it matters.

Specifications and Large Language Models (LLMs) are like the recipe and the chef. Give the same application specification to another LLM, and it will produce a materially different implementation.

Recently, at a hackathon, I saw a team of five write a specification for a timesheet application, and then go off to build something out of it individually. Each participant had produced a different result, with distinct strengths and weaknesses. One gamified timesheets with an interface that looked like a video game. Another added a natural-language input. They all had the fundamental features you can expect, but each participant added a twist while jamming with the LLM.

LLMs will also add unstated assumptions, preferences, and conventions learned from their training data.

The mistake is treating a specification as a source of truth when it is actually a communication protocol between an author and an interpreter. If variation in interpretation is inevitable, how do we construct feedback mechanisms that make it observable and correctable when working with LLMs?