General LS Linear Fit Theory

For a given set of observation data, the general least-squares (LS) linear fit problem is to find a set of coefficients that fits the linear model, as shown in Equation A:

(A)

where

xij is the observed data contained in the observation matrix H,
n is the number of elements in the set of observed data and the number of rows of in H,
b is the set of coefficients that fit the linear model,
k is the number of coefficients.

The following equation defines the observation matrix H.

(B)

You can rewrite Equation A as the following equation.

Y = HB. (C)

The general LS linear fit model is a multiple linear regression model. A multiple linear regression model uses several variables, xi0, xi1, …, xik – 1, to predict one variable, yi.

In most analysis situations, you acquire more observation data than coefficients. Equation A might not yield all the coefficients in set B. The fit problem becomes to find the coefficient set B that minimizes the difference between the observed data yi and the predicted value zi. Equation D defines zi.

(D)

You can use the least chi-square plane method to find the solution set B that minimizes the quantity given by Equation E.

= |H0BY0|2 (E)

where hoij = (xiji), yoi = (yii)

for i = 0, 1, …, n – 1, and j = 0, 1, …, k – 1.

In Equation E, σi is the standard deviation. If the measurement errors are independent and normally distributed with constant standard deviation, σi = σ, Equation E also is the least-square estimation.

You can minimize χ2 from Equation E in the following ways:

Solving normal equations involves completing the following steps.

  1. Set the partial derivatives of χ2 to zero with respect to b0, b1, …, bk – 1, as shown by the following equations.
(F)
  1. Derive the equations in Equation F to the following equation form:
    H0TH0B = H0TY (G)


    where H0T is the transpose of H0.

Equations of the form given by Equation G are called normal equations of the least-square problems. You can solve them using LU or Cholesky factorization algorithms. However, the solution from the normal equations is susceptible to roundoff error.

The preferred method of minimizing χ2 is to find the least-square solution of equations. Equation H defines the form of the least-square solution of equations.

H0B = Y0 (H)

You can use QR or SVD factorization to find the solution set B for Equation H. For QR factorization, you can use the Householder algorithm, the Givens algorithm, or the Givens 2 algorithm, which also is known as the fast Givens algorithm. Different algorithms can give you different precision. In some cases, if one algorithm cannot solve the equation, another algorithm might solve it. You can try different algorithms to find the one best suited for the observation data.