Do As I Mean, Not What I Say

I had a classification task that was perfectly suited for a large language model: text only input, self-contained guidance with only a few edge cases that could easily be overcome with few-shot examples.

Up until this point I had only interacted with LLMs via chat interfaces. Using an LLM to convert infinite variations of user input into a trusted, structured classification, ~500,000 times per day, was new to me.

My first prompt performed fine, but a far cry from the consistency and quality I needed. I began to consume model provider’s documentation⁰⁰ Not random blue-check-influencer-vibe-anecdotes 17 minutes after a model is released. People be saying things so definitively. Like man I think it depends. on how to approach designing a prompt. It was interesting to see where guidance converged and conflicted across providers.

Revisiting the prompt I was impressed it had worked at all. What had originally felt like a natural, clear request was in reality vague, contradictory and open to interpretation.

I found the iteration process to be highly engaging. Clarifying what I actually wanted. Making the subjective objective. Cutting the contradictions. Fast forward to 02:15am: 99% performance against my benchmark. Even so I still felt it could be taken to a more extreme level of refinement, but I was diminishing faster than the returns.

Returning to work with all I had learned, I became acutely aware of how many questions I asked, and were asked of me, that had many of the failings of my first prompt. This followed me home, where imprecision hides best. Years of shared context lets two people believe they are communicating when they are mostly just compensating for each other. I noticed it most with my children. We don’t yet have a shared context window to fill in the gaps. When they get it “wrong”, it’s rarely their fault. They mostly followed what I said, not what I meant, and filled the space between with whatever made sense to them. Often with something more creative and whimsical than I had in mind.

I dare wonder how many marriages have broken, projects failed to deliver, and expectations go unmet because an individual, couple, or corporation believed they were on the same page. Stranger yet is how much succeeds while being on different books, let alone pages. Maybe those “failures” are the collateral of having a world where not everything has to be said.

All these words, yet day to day I still succumb to the default human wiring. Assuming. What persists is a detector for when clarity makes a difference, and the habit of acting on it. My personal AGI will be a model that knows me so well I get XML packed prompt performance for the price of a vague one liner. In the mean time, the current frontier models have taught me to cherish the rare fleeting moments of human <> human alignment, and gratitude that humans don’t need to read an AGENTS.md before every interaction.