Least Squares Approximation

The method of least squares assumes that the best-fit curve of a given type is the curve that has the minimal sum of the deviations (offsets) squared (least square error) from a given set of data. The sum of the squares of the offsets is used instead of the offset absolute values because this allows the residuals to be treated as a continuous differential quantity.

The linear least squares fitting technique is the simplest and most commonly used form of linear regression and provides a solution to the problem of finding the best fitting straight line through a set of points.

Algorithm

Suppose the data points are (x₁, y₁), (x₂, y₂),…, (x_n, y_n) where x is the independent variable and y is the dependent variable. The fitting curve f(x) has the deviation (error) d from each data point, i.e. d₁= y₁- f(x₁), d₂= y₂- f(x₂),…d_n= y_n- f(x_n).

According to the method of least squares, the best fitting curve has the property that,

= a minimum

For a linear fit,

f(a, b) = a + bx

These lead to the equations,

In matrix form, the above equations become

Values of a and b can now be computed by taking matrix inverse.

Another form of representation is using sums of squares,

cov(x,y) is the covariance and and are variances.

In terms of sum of squares, the regression coefficient b is given by,

and a is given in terms of b using

os_num_least_squares class provides the capability to fit a collection of points onto a straight line using least squares approximation techniques. The class has various methods to calculate the scope, intercept, and other statistical quantities of such approximation.

The example code below shows how to use various methods supported by the os_least_squares class.

Example <ospace/math/examples/leastsq.cpp>


  #include <ospace/math.h>


int
main()
  {
  os_math_toolkit math_init;

  double vec_data1[ 5 ] = { 1, 2, 1, 3, 4 };
  double vec_data2[ 5 ] = { 3, 1, 3, 5, 8 };

  os_num_vector< double > vec_x( 5 );
  os_num_vector< double > vec_y( 5 );

  for ( int i = 0; i < 5; i++ )
    {
    vec_x[ i ] = vec_data1[ i ];
    vec_y[ i ] = vec_data2[ i ];
    }

  os_least_squares lsq( vec_x, vec_y );

  cout << "x = " << lsq.compute_x( 2 ) << endl;
  cout << "y = " << lsq.compute_y( 2 ) << endl;
  cout << "corr coeff = " << lsq.corr_coeff() << endl;
  cout << "intercept = " << lsq.intercept() << endl;
  cout << "intercept sdev = " << lsq.intercept_sdev() << endl;
  cout << "slope = " << lsq.slope() << endl;
  cout << "slope sdev = " << lsq.slope_sdev() << endl;


  return 0;
  }