Least Squares Solver using Normal Equations

Find the best approximate solution to $A \mathbf{x} \approx \mathbf{b}$ using the Normal Equations.
Solves $A^TA\mathbf{x} = A^T\mathbf{b}$ to minimize $\| A \mathbf{x} - \mathbf{b} \|_2$.

The Normal Equations provide a direct method for solving least squares problems:

$ A^TA\mathbf{x} = A^T\mathbf{b} $

Derivation:

Minimizing $\| A\mathbf{x} - \mathbf{b} \|_2^2$ requires the gradient to be zero:

$ \nabla_{\mathbf{x}} \| A\mathbf{x} - \mathbf{b} \|^2 = 2A^T(A\mathbf{x} - \mathbf{b}) = \mathbf{0} $

Key Points:

  • When $A$ has full column rank, $A^TA$ is invertible
  • Simple and direct approach for small to medium problems
  • Less numerically stable than QR for ill-conditioned matrices
Equations $m$: Variables $n$:
Display Values as:
© 2025 Shelvean Kapita: kapita@tamu.edu
All code released under the MIT License.
Last modified: August 5, 2025