Skip to main content

What 3,000 Product Reviews Taught Me About Trusting Star Ratings

 



I spent the last stretch of time buried in 3,000 customer reviews, trying to answer a question that sounds simple but turned out not to be: when a customer leaves a 3-star review, what are they actually telling you?

The short answer — not what you'd think. And the project I built around that question ended up teaching me more about designing trustworthy AI tools than about sentiment analysis itself. I'm sharing the whole thing, code included, here: github.com/dshah-ds/customer-sentiment-review-automation

Why I started digging into this

I think most of us, without really deciding to, treat star ratings as a shorthand for how a customer feels. 5 stars, thrilled. 1 star, furious. 3 stars... whatever, average, nothing to see here.

I wanted to check that assumption against the actual words customers used, not just the number they clicked. So I took a dataset of 3,000 reviews and ran the review text through a sentiment scoring tool, completely separate from the star rating, and then compared the two.

They disagreed almost half the time. 45% of reviews had a star rating that didn't match the tone of what the customer actually wrote. And the 3-star group was the biggest offender — 78% of those "neutral" reviews were, in plain English, positive. Customers who liked the product but didn't feel strongly enough to hit the top rating.

That also means there's a smaller group hiding in that same 3-star bucket who are genuinely unhappy but never bothered to rate it lower. If your process is "escalate the 1 and 2 star reviews," you're walking right past them.

Not all "unhappy" looks the same

The next thing I wanted to understand was whether Positive, Neutral, and Negative reviews were really just different points on the same line, or if they were actually different kinds of feedback that deserved different responses.

So I split the reviews into three groups and measured a few things about each one: how positive or negative the language was, how "urgent" it sounded (I counted words like broken, defective, refund, immediately), how long the reviews were, and which words showed up disproportionately in each group.

What I found made each group feel like a real, distinct type of customer rather than a data label:

  • Negative reviews were short and specific, and had a real urgency signal attached — customers who already knew exactly what was wrong and what they wanted done about it.
  • Neutral reviews had zero urgency, on average, and their own vocabulary entirely — "okay," "fine," "decent," "basic." Not angry, not thrilled, just unmoved.
  • Positive reviews were the most detailed and enthusiastic of the three, full of specific praise.

I ran the numbers to make sure this wasn't just my imagination — using a method called cosine similarity, which measures how alike two groups of text really are. All three groups came back clearly distinct from each other. They're not shades of the same feeling. They're three separate populations that need three separate responses.

Then I built the thing — and immediately found a flaw in it

The last part of the project was turning all of this into something usable: a small tool where you paste in a review and it tells you the sentiment, an urgency score, the likely themes, and even drafts a reply using Google's Gemini model that a support person could send.

While testing it, I found a bug I actually think is the most valuable part of the whole project.

One review said a product was "decent for the money." My tool flagged it as a Refund Request. Why? Because the word "money" is common in real refund complaints ("I want my money back"), so it was on the watch-list — but here it just meant "good value for the price." No refund involved anywhere.

I could have quietly fixed the keyword list and moved on like it never happened. Instead I left it in and wrote it up, because it's a small, honest example of exactly why I built this as a draft-and-review tool instead of a fully automatic one. A person glancing at that flagged review would catch the mistake in about two seconds. A system that skipped the human and auto-responded wouldn't — and would have treated a satisfied customer like an angry one.

That's the part of the project I'm proudest of, honestly. Not the parts that worked — the part where something broke in a way that made the argument for careful design better than any slide deck could.

If you want to poke around

Everything — the notebook, the three write-ups (insight generation, segment comparison, and the deployed tool), and my own reflection on what I'd change for a real production version — is up on GitHub:

github.com/dshah-ds/customer-sentiment-review-automation

I'd love to hear from a couple of different directions on this. If you've ever worked in support or read a pile of customer reviews for your own product, does that 45% mismatch number sound about right, or higher/lower than what you've noticed? And if data or AI work is newer territory for you, tell me what part of this you'd want unpacked further — happy to write a follow-up post on whichever piece people are most curious about.

Comments

Post a Comment

Popular posts from this blog

The First Notebook

Somewhere between where I was and where I am going! Welcome to One Notebook at a Time. Every data project starts the same way. A blank notebook. A first cell. No idea yet what the data is going to say, or whether the model is going to work, or whether the whole approach is wrong and you'll scrap it by Tuesday. You just start. This blog is that first cell. Why "One Notebook at a Time" I picked this name because it's literally true on two levels. If you're a data person, you already know what a notebook is — Jupyter, Colab, whatever your tool of choice. It's where the real thinking happens. Not the polished dashboard, not the final slide deck — the notebook. Messy cells, half-finished thoughts, a graph that didn't work followed by one that finally did. But it's also true in the older sense of the word. A notebook is just where you write things down before you know what they're going to become. That's what this space is for. One notebook — one pos...
Post a Comment
Read more

The Line of Best Fit Isn't "Best" — It's a Decision What I wish someone had told me before I ran my first regression model.

  Same data. Different definition of "best." Different line." Here's something that took me longer than I'd like to admit to figure out about linear regression. That line through your scatter plot — the one everyone calls the "line of best fit" — isn't objectively the best anything. It's the best fit for one very specific definition of the word "best." Swap that definition, and you get a completely different line. Equally valid mathematically. Just built on a different set of assumptions. Once I understood that, I started looking at every machine learning model differently. What Everyone Gets Taught (And What Gets Left Out) The standard introduction to linear regression goes something like this: draw a line through your data that minimises the total error between your predictions and the actual values. Square those errors, add them up, minimise the sum. Done. This is Ordinary Least Squares (OLS) — the engine behind most introductory r...
Post a Comment
Read more

The Network That Learns Who Comes Back

It's been a minute since I last posted here — most of that gap was spent heads-down on a project I just wrapped up, so let's catch up. Where a heartbeat becomes a data point — teaching a neural network to see risk before it happens. If you've followed along before, you know this space usually lives in supervised learning — regression, classification, the more "classical" side of the toolkit. This post is a deliberate left turn into neural networks , the first one I've built and documented fully end-to-end. I expect this thread to keep rotating: NLP is probably next, and I'll likely circle back to supervised or unsupervised learning again after that. Think of it as this notebook's current chapter, not a permanent pivot. The setup The project: predict whether a patient will be readmitted to hospital within 30 days of discharge, using a 5,000-row dataset of vitals, prior admission history, chronic conditions, and a handful of administrative fields. It...
Post a Comment
Read more