What Least Squares Assumes About Your Data

In my linear algebra course in college I learned about least squares regression, where it wasn't very motivated outside of "we square the error because it makes error non-negative, really penalizes outliers, and it's easy to compute".

Recently, I learned about maximum likelihood estimation, where we pick parameters for a model based on whatever maximizes the likelihood of the observed data. For example, if we have a random variable 

  Y = a X + b + E
where
  X ~ U(0, 1)         (X uniformly sampled from [0, 1])
  E ~ N(0, s^2)       (error sampled from a Gaussian distribution)

Given some set of samples (X1, Y1), (X2, Y2), ..., (Xn, Yn), omitting some constants for clarity, we can find the parameters (a, b) that maximize the likelihood via: 

  argmax[a,b] of product[i=1...n] of exp(-((Yi - a Xi - b) / s)^2)
                                           { taking logarithms, an increasing function }
= argmax[a,b] of sum[i=1...n] of -((Yi - a Xi - b) / s)^2
                                           { taking out the constant -s^2 term }
= argmin[a,b] of sum[i=1...n] of (Yi - (a Xi + b))^2

Which is just a least squares fit.

So, least squares can be interpreted as maximizing likelihood, assuming that the error is drawn from a normal distribution. Assuming other distributions yield other maximum likely hood estimators. For example, assuming Laplace-distributed errors yields an estimator that minimizes the sum of absolute errors instead of squared errors.

A gap in knowledge that I had for the longest time was understanding what a model is. I had some fuzzy idea that a model is a simplified version of reality, that follows strict rules that let us do some calculations about the model. In retrospect I was confusing models with abstractions.

I now think of a model as a belief, a claim regarding the process that generates some data set.

Once you commit to believing that your model is true, you can derive exact quantities: probabilities, expected values, MLEs, etc. Then, you can apply those results "blindly" to the real world (the model is true, after all).

If the estimations fail, we don't just patch them with ad-hoc fixes. Instead, we push those changes back into the model. We ask: what change to the model would yield the change in estimator that I see is needed? Perhaps the answer then is adding a new parameter, who knows.

There's even the extreme version of this: In practice we often come up with solutions or rules of thumb without any model behind them at all. But we can ask: for what model or models would this rule be the exact optimal solution? That alone can point to cases where the rule would break down.

Thinking about real-world problem solving this way feels less like just picking up tricks and more like a systematic (scientific, even) process of modeling -> testing -> refining models.

Comments

Post a Comment

Popular posts from this blog

[EN] Writing a Compiler is Surprisingly Easy (part 1)

Part 1: expressions, arithmetic Part 2: statements, control flow Ever since I was a teenager I wanted to create my own systems programming language. Such a programming language would certainly have to be compiled to native code, which meant I'd have to write a compiler. Even though I managed to write several half-working parsers, I'd always fail at the stage of generating assembly code, as the task turned too complex. In this blog I intend to show my teenage self how writing a code generator is, in fact, not complex at all, and it can be fully done in a couple of weekends. (As long as we make some simplifying assumptions) I will that assume you already know that representing programs within a program is surprisingly easy Where do we even start? The goal today is to translate a high level language to x86-64 machine code. Our high-level language will have integer literals, variables, negation, and addition. To make our job simpler, we will translate the hig...
Read more

[EN] Writing a Compiler is Surprisingly Easy (part 2)

Part 1: expressions, arithmetic Part 2: statements, control flow In my first few attempts at writing a compiler, a big hurdle was compiling control flow (conditionals, loops, etc). I think this stemmed from the fact that the a lot of the common advice that's floating around the internet is for writing an optimizing compiler, so I would hear things like 'static single-assignment', 'control flow graphs', 'phi-nodes' and the like, and get lost trying to take it all in. Because of such things, my first serious attempt at implementing a programming language ended up begin a tree-walking interpreter, and it was several years until I dared write a compiler. I'd like to show that all these things, while interesting and valuable in their fields of application, are not required to write a useful compiler. In particular, we'll see at the end that even a naive compilation strategy will handily beat a fast tree-walking interpreter in speed. State...
Read more

[EN] Representing Programs Within Programs is Surprisingly Easy

Suppose that you want to create a brand new programming language that will change the world. Or that you want to auto-formatter that gets indentation just right . Or maybe you want to write a pre-processor for an existing language, to hack some missing feature into it . Whatever it may be, there are many reasons to write programs that manipulate other programs as data and the first step is to figure out how you are going to represent that data within the program. My goal in this article is to explain one of the most standard approaches (Abstract Syntax Trees) and show how they are actually really easy to understand and implement. This article has many code examples in the C language. What kind of language are we defining? Let's start with a simple example: a language with variables, numbers, and addition. Note that this language does not have statements, but only expressions that evaluate to simple values. This means that we will be able to represent expressions such as...
Read more