The holy grail of marketing is customer insight. New and exclusive information about your customer that delivers a genuine competitive advantage, enabling you to enhance your offer in ways that feed straight through to increased sales.

But recognizing the importance of insights is the easy part, actually finding them can be tricky. Many businesses put significant time and effort into looking for customer insights, only for their search to end in frustration and failure, much like the quest for the actual holy grail.

But don’t despair, there is a reason to be cheerful. Unlike the fictional holy grail, which is an imagined artefact in Arthurian legend, customer insights, while quite rare, are very much real and attainable. You just need to take the right approach to find them, which is what this article is about.

We should start with an obvious yet fundamental question; what exactly is an insight? Well, I’m glad you asked.

Insights defined

Formally, ‘insight’ is defined as “gaining a new, accurate and deep understanding of someone or something”, which, though precise, is insufficient. We need a more refined and specific definition.

I prefer the one proposed by Paul Laughlin, in the Institute of Direct and Digital Marketing’s Journal; “Insight is a non-obvious understanding about your customer, which, if acted upon, has the potential to change their behaviour for mutual benefit”.

Laughlin’s definition gives a prospective insight four hurdles to jump over before it can gallop across the finish line.

Hurdle 1 – Non-obvious (the “no shit” hurdle)

An insight must clearly be something new, but novelty alone isn’t sufficient. It also needs to be fairly surprising and unexpected. Counterintuitive, even. An ‘insight’ that is easily discoverable or clearly predictable or widely anticipated, isn’t really an insight at all.

Hurdle 2 – Actionable (the “so what?” hurdle)

An insight must be capable of generating a testable hypothesis. Discovering that Kevin is the most common name among men who buy socks online on Thursdays might be sufficiently non-obvious to jump over the first hurdle but that’s about all. If you can’t find the ‘why’ behind a non-obvious data point you can’t design a ‘what if’ type of experiment.

Hurdle 3 – Behaviour changing (the “lead, don’t follow” hurdle)

An insight must be capable of changing a customer’s behaviour. Finding out that customers are already behaving in a certain way and then adapting your offer to follow them isn’t an insight, it’s playing catch-up.

Hurdle 4 – Mutually beneficial (the “win-win” hurdle)

An insight must create an opportunity for you to enhance your customer’s experience in a way that increases (or at least maintains) your long term profitability.

This can be a challenge, especially for those insights that come disguised as threats. A bit of creativity and effort is often needed to get over this last hurdle and on to the finish line.

The standards required for an insight to clear these four hurdles are strict and demanding. Which explains why those that make it to the end are quite rare and exceptionally powerful. And why it is worth putting the effort in to find them.

Assuming the strictness of the definition, and its attendant high failure rate hasn’t put you off, now would be a good time to focus on how we go about finding insights.  

Discovering Insights

First, a caveat. The use of the verb ‘discover’ has the potential to mislead. It paints a picture of insights being hidden somewhere, fully formed and complete, just waiting to be found. Which is rarely the case.

When I use ‘discover’ I mean it in the sense of Charles Darwin discovering evolution, not in the sense of Captain Cook discovering Australia. If that helps.

Insights are the product of combining and synthesizing facts and observations. They are a construct of thoughtful analysis. evidence gathering, intuition, and experimentation. You develop them more than you find them.

Insight hunting is as much art as science. Consequently, no single formal process or methodology exists that works for all businesses.

But that doesn’t mean we can’t put a little structure around the idea.

The What, the Why and the wow

Think of insights as sitting on the end of a three-step progression of data, knowledge and insight. Linked together by what why and wow.

Which is not to suggest this is a formal three-step process you have to follow. Some of the biggest insights have revealed themselves in a serendipitous eureka moment without any formal process, but it will help to categorize things as we develop the idea.

We have already defined insight, so let’s briefly turn to data and knowledge.

Data

Data are facts, statistics and disparate pieces of information. They could take the form of published market data or proprietary market research. They could be intelligence from your internal information systems. They could be something you’ve heard. They could be something you’ve seen. They could be something you’ve read.

Data, in whatever form, are simply the raw materials. If an insight is a cake then data are the eggs, butter, sugar and flour. They have the potential to become something desirable but require some mixing and a degree of knowledge and skill to realize their potential.

Knowledge

Knowledge is gained by analysing data to obtain a deeper grasp of the subject you are studying or an answer to the question you are asking.

Knowledge is not, in itself, an insight. This is arguable (at least judging by the number of arguments I have had about it) but my rationale is this; Knowledge often answers the question ‘what?’ while insights often answer the question ‘why?’. Not always, not universally, but frequently enough.

“Why?’ is the key that unlocks knowledge and turns it into insight.

How to find the ‘why’ – a five-step process

We have already established that discovering insights is more à la carte than a set menu. So, what follows is a method rather than the method. However, it is an approach and has delivered results across multiple industries and markets. I call it ‘contextual enquiry’ (although I’m not very good at naming things according to my children, Trenchant and Biscuit)

Step one – Assemble a team

Insight hunting is not a solitary endeavour. Obviously, a team will lighten the burden and share the workload but that is not its primary function. A diverse team drawn from different departments will give you a range of perspectives and viewpoints.

Insights require a synthesizing of information and blending of ideas and opinions. Conventional thinking can be a handicap, so find an eclectic mix of six to eight people with a taste for adventure, open minds, a willingness to ignore convention and sufficient time to commit.

Step two – What do we know? What do we wish we knew?

Gather your team of explorers and brainstorm a long list that captures everything you know, or would like to know, about your customer.

Don’t limit yourself. You want to know everything that relates to their job or the outcome they are trying to achieve. You want to know about their customers, their environment, their needs, how they use your product and so on.

Use the classic interrogatives of what, why, who, where, when, and how.

It takes a significant amount of time and effort to develop a comprehensive list and isn’t the sort of thing you do in a half-hour brainstorm. Do it over two meetings with a week or so in-between to allow a little time for ideas to percolate.

At the end of this process the items on your list will end up in one of three categories; things we are confident that we know, things we believe to be true (and need to validate) and things we don’t know (and need to find out).

Step three – validate and refine

The last internal step, before we get to talking with customers, is to fill in the blanks in our knowledge using available data. We are trying to remove as much doubt as possible by turning as many of the ‘we believe this to be true’ statements and the ‘we don’t know questions’ into the ‘we are confident we know’ category.

This stage can take anywhere from a few days to a month or longer. The rule is to not talk to customers (yet) and only fill in the blanks with internal data or available secondary data.

When you are finished you have a baseline of knowledge that becomes the blueprint for the next critical phase.

Step four – the contextual enquiry bit

The most reliable way to find latent, unstated needs is to observe your customer using the product in the context of their day to day environment. And then asking relevant and probing questions.

This obviously has some limitations depending on the nature of the product, but a bit of creativity can make it work in most situations. The most critical elements are your ability to subtly observe and effectively question.

A) Subtly observe

People act differently when they know they are being observed. Leaning over their shoulder with a clipboard in hand will not always give you an accurate interpretation of reality.

Subtle observation requires you to be ninja-like. Be unobtrusive and inconspicuous, keep it casual, don’t interrupt with too many questions. Just watch and learn.

It should go without saying that any observation should be carried out ethically, consensually and legally. Don’t be hiding in the bushes or peering through windows. Do that on your own time.

B) Effectively question

Based on your earlier internal efforts you will have lines of enquiry pre-planned. In particular, finding answers to those questions that are still in the ‘we don’t know’ category. But effective questioning should dynamic and be informed by what you have just observed.

There is no definitive list of questions, but a taster menu may include ‘Why did you do that?’, ‘What is the effect of this?’, ‘How often does it happen?’ ‘Why do you do this?’, ‘what would you change?’, ‘Why is that important?’, ‘How do you feel when this happens?’, …and so on.

Step five – What if…?

Your fieldwork should involve all of your team individually visiting more than one customer. I realise that in this microwavable, instant gratification, pot noodle, ‘just do it’, world we live in, that sounds anachronistically quaint. But remember, we are treasure hunting. We are not looking for incremental improvement. Insights typically require some hard yards; it isn’t effort-free.

Regroup as a team and share findings. Hypothesize an insight or two (or three or four if you’re showing off). Check them against the four hurdle criteria from earlier. Prioritize them and explore what could be done to enhance your offer. Ask what if… a lot.

To take your insight further you will need a whole new set of tools; Design of experiments (DOE), minimally viable product (MVP), new product introduction (NPI), and other three-letter acronyms (TLA), that go beyond the scope of this article (SoTA).

Conclusion

Genuine customer insights are as rare as they are powerful. The path to finding them is littered with dangerous traps and numerous ways to fail. It requires significant effort and focus. Your quest will take you on an exciting journey of discovery, with a few setbacks along the way, before you eventually reach your goal.

But it a voyage worth taking. The treasure at the end of your journey will more than compensate for the effort required to get there.

So, pop on your wide-brimmed fedora, grab your whip, throw a rope over your shoulder, and unleash the spirit of Indiana Jones. Your adventure is about to begin.

Faint heart never found great insight.