The Path Worn Through the Grass: Observation and Metrics

A university campus, freshly built. Paved paths run between the buildings at neat right angles. Within a month there's a brown line cutting diagonally across the lawn, exactly where nobody planned a path and everybody walks.

Those tracks have a name — desire paths — and some architects now delay paving for a season, let people wear the routes in, and then lay stone over the lines that appeared. That single decision contains most of what this article is about: the difference between the route you designed and the route in use, and the two very different tools that reveal each one.

The gap between saying and doing

The previous article was about asking well. This one starts with the limits of asking at all.

People misreport their own behaviour, consistently and innocently — how often they check their phone, how long a task takes, how they organise their files. Not because they're evasive, but because behaviour is mostly automatic and memory summarises it into a tidy story.

So the second research tool is simpler than the first: stop asking and watch.

Hugh Beyer and Karen Holtzblatt formalised this in 1997 as contextual inquiry, and its central image is the master and the apprentice. You don't interview the master in a meeting room; you stand next to them while they work and ask about what's happening as it happens. Four principles hold it together:

Context — go where the work actually happens, not where it's convenient to discuss it.
Partnership — you're the apprentice; they're teaching you their job. You don't lead, and you don't test them.
Interpretation — you say your reading aloud ("so this bit is a nuisance because…") and let them correct you. Their corrections are the data.
Focus — you have a question you're there to answer, otherwise you'll come back with eight hours of everything.

What to look for

Watching is unstructured, so it helps to hunt specific things.

Workarounds. The spreadsheet beside your product. The paper checklist. The second browser profile. The naming convention with a date in it. Every workaround is a feature request already written and already validated by someone spending their own time on it.

Hesitation. The half-second pause before a click. Nobody reports it in an interview, because it doesn't survive as a memory. But it marks a decision that cost something, and decisions that cost something are exactly where the design failed.

The wrong order. People doing steps in an order your flow didn't anticipate, and reverse-engineering their way back.

Copy and paste. Data being carried by hand between two systems is a missing integration wearing a disguise.

What they never touch. Whole regions of your interface with no traffic at all. In a demo people click everything; in real work they use six things.

The words they use. They call it a "job," you call it a "project," the code calls it an "entity." Whoever loses this argument, it shouldn't be the user.

What numbers are for

Observation shows you a handful of people in depth. Analytics shows you everyone, shallowly. They answer different questions, and mixing them up is the most common analytical mistake in product work.

Quantitative data answers what, where, how many, how often. Drop-off between step 3 and step 4. Median time to first value. Share of sessions on mobile. Search queries with no results — an underrated goldmine, since each one is a person telling you a word your product doesn't know.

Qualitative data answers why. And nothing else does. A funnel showing 60% loss at the address step is a fact; whether that's confusion, distrust, a missing payment method or a broken field is a different investigation.

The useful pairing: quantitative finds where to look, qualitative explains what you found. Run them in that order and neither is wasted.

A few measures worth knowing by name:

Task success rate — the share of people who complete a defined task. Blunt, and the most honest single number in usability.
Time on task — useful, but not monotone: faster is better for a routine action and meaningless for an exploratory one.
Error rate and recovery — how often people go wrong, and whether they get back.
Retention — the only metric that can't be faked by a good landing page.

Google's HEART framework organises these sensibly: Happiness, Engagement, Adoption, Retention, Task success — each turned into concrete signals and metrics via the goals–signals–metrics chain. Its real value is the middle step: forcing you to say what observable signal would indicate the goal, before you pick a number.

Numbers that lie

Metrics fail in specific, repeatable ways. Learn the shapes.

Vanity metrics. Totals that only go up — registered users, page views, downloads. They can't go down, so they can't tell you anything is wrong.

Averages over two populations. An average session of four minutes may be two groups: people who leave in ten seconds and people who stay ten minutes. The average describes nobody. Look at distributions.

Survivorship bias. Your analytics measure the people still there. The ones your design drove away contribute nothing, forever — which is why a product can look healthier as it gets narrower.

Goodhart's law. When a measure becomes a target, it stops being a good measure. Optimise for time-in-app and you'll get an app that's hard to leave, which is not the same as an app worth using.

Confusing an event with an intention. A click is not approval, a repeat visit is not loyalty, and a long session can mean either delight or being lost.

About surveys

Surveys look like the cheap option and are in fact the hardest instrument here to use honestly. A question can lean without you noticing ("How useful did you find the new dashboard?" has already assumed useful). Scales get interpreted differently by different people. And the people who answer are systematically the most enthusiastic and the most annoyed, with the vast indifferent middle silent.

Two standard instruments, honestly labelled:

SUS — the System Usability Scale, ten fixed questions, devised by John Brooke in 1986, scored out of 100 with roughly 68 as an average. Its strength is precisely that you don't write the questions, so you can compare across versions and products.

NPS — "how likely are you to recommend this," collapsed into one number. Popular with executives, weakly related to actual behaviour, and it tells you nothing about what to change. If it's mandated, treat the free-text comment underneath as the actual data.

Use surveys for what they're good at: measuring something you already understand, across many people, repeatedly over time. Never use one to discover a problem you haven't seen yet.

In practice

Sit behind someone for an hour. With their real work, not a task you invented. Say almost nothing. Write down every hesitation and every workaround.

Read your empty search queries. Words with no results are your users' vocabulary colliding with yours. Ten minutes for a list of missing labels and missing features.

Turn one number into a question. Take the worst drop in your funnel and go find three people who dropped there. Don't theorise about the number; go and see it.

Look at distributions, not averages. For every average you rely on, look at the actual spread once. It will change at least one of your beliefs.

Write the signal before the metric. Before choosing a number, finish the sentence: "if this is working, we'd see people…" Then pick something that measures that sentence.

Photograph the workaround. When you find a spreadsheet someone built to survive your product, capture it. It's the most persuasive artefact you will ever bring to a product meeting.

Check yourself

Close the article and answer in your own words:

What's a desire path, and what's the equivalent in a digital product?
Name the four principles of contextual inquiry and what each one prevents.
Why is a workaround more valuable than a feature request?
Which questions can analytics answer, and which can it never answer?
Explain survivorship bias using your own product as the example.
What is Goodhart's law, and give a metric it could ruin.
What are surveys actually good for, and what should you never use them for?

In short

Desire paths: what people do diverges from what was designed, and the divergence is data rather than misbehaviour.
People misreport their own behaviour innocently. Watching beats asking for anything habitual.
Contextual inquiry: go to the work, act as apprentice, say your interpretation aloud, keep a focus.
Hunt for workarounds, hesitations, wrong order, copy-paste, untouched regions and the user's own vocabulary.
Quantitative answers what and where; qualitative answers why. Numbers find the spot, people explain it.
Metrics fail as vanity totals, averages over two populations, survivorship bias, Goodhart's law, and events mistaken for intentions.
Surveys measure the known at scale; they don't discover the unknown. SUS is comparable; NPS mostly isn't useful for design.