Skip to main content

The Line of Best Fit Isn't "Best" — It's a Decision What I wish someone had told me before I ran my first regression model.

 

Same data. Different definition of "best." Different line."

Here's something that took me longer than I'd like to admit to figure out about linear regression.

That line through your scatter plot — the one everyone calls the "line of best fit" — isn't objectively the best anything. It's the best fit for one very specific definition of the word "best."

Swap that definition, and you get a completely different line. Equally valid mathematically. Just built on a different set of assumptions.

Once I understood that, I started looking at every machine learning model differently.

What Everyone Gets Taught (And What Gets Left Out)

The standard introduction to linear regression goes something like this: draw a line through your data that minimises the total error between your predictions and the actual values. Square those errors, add them up, minimise the sum. Done.

This is Ordinary Least Squares (OLS) — the engine behind most introductory regression models, and one of the most well-understood methods in all of statistics.

But here's the question that almost never gets asked in that introductory lesson: why squared errors specifically?

The Hidden Decision Inside the Formula

When you square your errors instead of, say, taking their absolute values, something specific happens to your model's behaviour.

Squaring is not neutral. It amplifies large errors disproportionately — an error of 20 doesn't just become twice as costly as an error of 10, it becomes four times as costly. An error of 100 becomes one hundred times as costly as an error of 10.

The consequence: your model becomes heavily influenced by outliers. Data points sitting far from the main cluster pull the line toward them with outsized force, because their large errors — once squared — dominate the entire calculation.

This is a design decision baked into the algorithm. Whoever chose to minimise squared errors was implicitly deciding: large mistakes matter much more than small ones. That's often a reasonable position. But it's a position, not a mathematical law.

A Different Choice: Mean Absolute Error

Mean Absolute Error (MAE) takes the absolute value of each error instead of squaring it. An error of 20 is simply twice as costly as an error of 10 — proportional, intuitive, straightforward.

The result is a model that treats outliers more evenly. It doesn't get pulled toward unusual data points with the same force that OLS does.

Technically speaking:

  • OLS finds the conditional mean — the line passing through the average of y for every value of x
  • MAE finds the conditional median — the line passing through the middle value of y for every value of x

Neither is universally better. They answer slightly different questions. The right choice depends entirely on your data and what you're trying to do with the model.

What I Saw in a Real Dataset

I ran into this exact issue working through the Housing Dataset — predicting house sale prices across hundreds of properties with dozens of features.

The OLS model performed well on typical houses. But a small number of high-value outlier properties — unusual homes with sale prices far above the typical range — were pulling the regression line in ways that quietly hurt performance everywhere else. Large squared errors from those few properties were carrying a weight in the loss function that the raw numbers didn't justify.

It's the kind of thing you don't notice if you're only looking at your overall R² score. You have to dig into the residuals, look at where your model is struggling, and ask why.

The answer, in this case, traced all the way back to the choice of loss function.

The Real Lesson Underneath All of This

Loss functions are choices. Assumptions are choices. Even the word "best" in "best fit" is a choice — best for a specific metric, under specific conditions, for a specific purpose.

Every machine learning algorithm you will ever use has decisions like this buried inside it. Decisions made by researchers and practitioners before you, embedded into the method, and handed down as defaults.

The job of a thoughtful data scientist isn't just to run the default and report the output. It's to understand what choices were made, why they were made, and when a different choice might serve your problem better.

Linear regression, understood at this level, isn't just a simple algorithm. It's a masterclass in how to ask better questions about every model that follows it.

The Short Version

OLS minimizes squared errors — a deliberate design choice that makes your model sensitive to outliers. MAE minimizes absolute errors — a different choice, more robust to unusual data points. The "line of best fit" is only best for the definition of "best" embedded in your loss function. Knowing that gives you the power to question it — and change it when your data calls for something different.

What's Next

This is Notebook #2 in an ongoing series working through data science from the foundations up — not as a textbook would, but as I'm actually learning and applying it.

The next notebook could go deeper into regression, shift to something entirely different, or revisit something I touched on too briefly here. A working notebook follows the thinking, not a fixed syllabus.

Follow along so you don't miss wherever it goes next.

If this sparked a question or you see something worth pushing back on, drop it in the comments. This is a working notebook — disagreement and discussion are part of the process.

The code behind the Housing project referenced in this post is available on my GitHub — github.com/dshah-ds. Feel free to explore, fork, or reach out if you have questions about the implementation.

Comments

Post a Comment

Popular posts from this blog

The First Notebook

Somewhere between where I was and where I am going! Welcome to One Notebook at a Time. Every data project starts the same way. A blank notebook. A first cell. No idea yet what the data is going to say, or whether the model is going to work, or whether the whole approach is wrong and you'll scrap it by Tuesday. You just start. This blog is that first cell. Why "One Notebook at a Time" I picked this name because it's literally true on two levels. If you're a data person, you already know what a notebook is — Jupyter, Colab, whatever your tool of choice. It's where the real thinking happens. Not the polished dashboard, not the final slide deck — the notebook. Messy cells, half-finished thoughts, a graph that didn't work followed by one that finally did. But it's also true in the older sense of the word. A notebook is just where you write things down before you know what they're going to become. That's what this space is for. One notebook — one pos...
Post a Comment
Read more

The Network That Learns Who Comes Back

It's been a minute since I last posted here — most of that gap was spent heads-down on a project I just wrapped up, so let's catch up. Where a heartbeat becomes a data point — teaching a neural network to see risk before it happens. If you've followed along before, you know this space usually lives in supervised learning — regression, classification, the more "classical" side of the toolkit. This post is a deliberate left turn into neural networks , the first one I've built and documented fully end-to-end. I expect this thread to keep rotating: NLP is probably next, and I'll likely circle back to supervised or unsupervised learning again after that. Think of it as this notebook's current chapter, not a permanent pivot. The setup The project: predict whether a patient will be readmitted to hospital within 30 days of discharge, using a 5,000-row dataset of vitals, prior admission history, chronic conditions, and a handful of administrative fields. It...
Post a Comment
Read more