Turning Feedback into Insights: A Smart Analysis Method

In today's data-driven world, organizations are sitting on a goldmine of information hidden within customer reviews, employee surveys, and support interactions. But how do you transform thousands of scattered comments into meaningful, actionable insights? Let me walk you through our approach to feedback analysis.

The challenge of feedback overload

Imagine having hundreds or thousands of feedback documents—app store reviews, employee survey responses, customer support messages—and trying to make sense of them manually. It's overwhelming, time-consuming, and prone to human bias. We developed a systematic methodology to solve this problem.

Our five-step feedback analysis process

1. Contextual span extraction

The feedback coming into the system can be anything from a few words to several paragraphs, so we need to get it into a more uniform shape. This is why our first step is to break down the incoming feedback into discrete chunks called "spans".

A key part of this step is to ensure each span retains meaning and context, we don't just want single words or ambiguous phrases.

Here's an example hotel review for Fawlty Towers:

★★☆☆☆ Absolute nightmare! Checked in and was immediately insulted by the manager, who seemed to have a personal vendetta against everyone. The room was dusty, the bed was uncomfortable, and I'm pretty sure the waiter was a bit... unhinged. Breakfast was a comedy of errors - eggs that looked like they'd been trampled, toast burnt to a crisp, and the owner kept muttering something about "not mentioning the war." Would not recommend.

The spans we'd extract from this might be:

• Checked in and was immediately insulted by the manager
• The room was dusty
• the bed was uncomfortable
• the waiter was a bit... unhinged
• Breakfast was a comedy of errors - eggs that looked like they'd been trampled, toast burned to a crisp
• the owner kept muttering something about "not mentioning the war."

2. Categorisation / coding, and sentiment analysis

A key piece of our platform is the ability to view insights about specific areas of your business. No two organisations are the same, so we have a unique tree of categories and subcategories for each one. We go through each span and match it up to one of the subcategories.

While we're categorising the spans, we also analyse the sentiment which will come in handy later on.

3. Generating "embeddings"

We convert the text from each span into mathematical vectors that capture semantic meaning. This allows us to compare and cluster feedback more intelligently than traditional keyword matching.

4. Clustering

Now we have a collection of categorised spans with their sentiment, and a mathematical representation of the text we can start grouping them together. We run this on all the spans in each subcategory to create clusters of spans which are discussing similar themes and topics. This identifies groupings of common themes across multiple responses, revealing patterns that might be missed by manual review.

5. Summarisation

For each cluster of feedback, we use a Large Language Model (LLM) to generate concise, meaningful summaries. This transforms raw data into digestible insights which the organisation can explore, discuss, and act on.

We now have a set of insights which speak directly to the organisation, and are anchored to the real feedback employees and customers are giving them.

Why This Matters

This approach lets organizations:

• Process thousands of feedback documents quickly
• Identify emerging trends and issues
• Jump directly from high level insights down to the explicit pieces of feedback which form them
• Understand customer and employee sentiment at scale
• Make data-driven decisions a rich qualitative source that was previously hard to use

Why are we sharing this, and what's next?

We're really happy to talk about our process in detail because we believe there is so much more still left to do. This year alone we've already made significant changes to our methodology.

We're continually perusing new ideas to improve our methodology, and use it in new ways. From sector wide competitor analysis, to improving the way organisations collect feedback in the first place - there's lots more to come!

Interested in transforming your feedback analysis? Let's chat.