Python:Fitting
This document contains examples of polynomial fitting, general linear regression, and nonlinear regression. In each section, there will be example code that may come in useful for later courses. The example code is based on the existence of a file in the same directory called Cantilever.dat that contains two columns of data - the first is an amount of mass (in kg) placed at the end of a beam and the second is a displacement, measured in inches, at the end of the beam. For EGR 103, this file is:
0.000000 0.005211
0.113510002 0.158707
0.227279999 0.31399
0.340790009 0.474619
0.455809998 0.636769
0.569320007 0.77989
0.683630005 0.936634
0.797140015 0.999986
Contents
Common Command Reference
All links below to NumPy v1.15 manual at NumPy v1.15 Manual; these commands show up in just about all the examples:
In the scripts below, common code has a regular background while code that differs from script to script will be highlighted in yellow.
Common Code
The processes for loading and manipulating the original data, calculating the statistics, and creating the graph are the same across the examples. To simplify codes, those processes should be contained in a separate file called fitting_common.py as given below:
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4
5 def get_beam_data():
6 # Load data from Cantilever.dat
7 beam_data = np.loadtxt("Cantilever.dat")
8 # Copy data from each column into new variables
9 mass = beam_data[:, 0].copy()
10 disp = beam_data[:, 1].copy()
11 # Convert mass to force
12 force = mass * 9.81
13 # Convert disp to meters
14 disp = (disp * 2.54) / 100
15
16 fmodel = np.linspace(np.min(force), np.max(force), 100)
17
18 # Return values
19 return force, disp, fmodel
20
21
22 def calc_stats(y, yhat, to_print=1):
23 # Calculate statistics
24 st = np.sum((y - np.mean(y)) ** 2)
25 sr = np.sum((y - yhat) ** 2)
26 r2 = (st - sr) / st
27 if to_print:
28 print("st: {:.3e}\nsr: {:.3e}\nr2: {:.3e}".format(st, sr, r2))
29 return st, sr, r2
30
31
32 def make_plot(x, y, yhat, xmodel, ymodel, fig_num=1):
33 fig, ax = plt.subplots(num=fig_num, clear=True)
34 ax.plot(x, y, "ko", label="Data")
35 ax.plot(x, yhat, "ks", label="Estimates", mfc="none")
36 ax.plot(xmodel, ymodel, "k-", label="Model")
37 ax.grid(True)
38 ax.legend(location="best")
39 fig.tight_layout()
40 return fig, ax
Polynomial Fitting
Polynomial fits are those where the dependent data is related to some set of integer powers of the independent variable. MATLAB's built-in polyfit command can determine the coefficients of a polynomial fit.
Specific Command References
All links below to NumPy v1.15 manual at NumPy v1.15 Manual
Example Code
In the example code below, n determines the order of the fit. Not much else would ever need to change.
1 # %% Import modules
2 import numpy as np
3 from fitting_common import *
4
5 # %% Load and manipulate data
6 x, y, xmodel = get_beam_data()
7
8 # %% Perform calculations
9 n = 1
10 p = np.polyfit(x, y, n)
11 print(p)
12
13
14
15
16
17
18
19
20 # %% Generate estimates and model
21 yhat = np.polyval(p, x)
22 ymodel = np.polyval(p, xmodel)
23
24 # %% Calculate statistics
25 calc_stats(y, yhat, 1)
26
27 # %% Generate plots
28 make_plot(x, y, yhat, xmodel, ymodel)
General Linear Regression
General linear regression involves finding some set of coefficients for fits that can be written as:
where the \(a_m\) are the coefficients of the fit and the \(\phi_m(x)\) are the specific functions of the independent variable that make up the fit.
Specific Command References
All links below to NumPy v1.15 manual at NumPy v1.15 Manual
Example Code
In the example code below, there is an example of a general linear fits of one variable. It is solving the same fit as given above, just in different way. Specifically it uses linear algebra to find the coefficients that minimize the sum of the squares of the estimate residuals for a general linear fit. In this code, variables ending in "v" explicitly need to be column vectors while variables ending in "e" can either be 1D arrays or 2D arrays (or lists).
Note - if you get an error about "TypeError: must be real number, not NoneType" then on line 15, replace None with -1.
1 # %% Import modules
2 import numpy as np
3 from fitting_common import *
4
5 # %% Load and manipulate data
6 x, y, xmodel = get_beam_data()
7
8 # %% Perform calculations
9 def yfun(xe, coefs):
10 return coefs[0] * xe**1 + coefs[1] * xe**0
11
12
13 # Reshape data for block matrices
14 xv = np.reshape(x, (-1, 1))
15 yv = np.reshape(y, (-1, 1))
16 phi_mat = np.block([[xv**1, xv**0]])
17 pvec = np.linalg.lstsq(phi_mat, yv, rcond=None)[0]
18 print(pvec)
19
20 # %% Generate estimates and model
21 yhat = yfun(x, pvec)
22 ymodel = yfun(xmodel, pvec)
23
24 # %% Calculate statistics
25 calc_stats(y, yhat, 1)
26
27 # %% Generate plots
28 make_plot(x, y, yhat, xmodel, ymodel)
Nonlinear Regression
Nonlinear regression is both more powerful and more sensitive than linear regression. For inherently nonlinear fits, it will also produce a better \(S_r\) value than linearization since the nonlinear regression process is minimizing the \(S_r\) of the actual data rather than that of the transformed values. The sensitivity comes into play as the optimization routine may find local minima versus global minima. A good starting guess will work wonders.
Specific Command References
The link below is to the SciPy v1.1.0 reference guide at SciPy
Steps for Nonlinear Regression
The critical parts of solving for the nonlinear regression involve defining the function, setting the initial conditions, and understanding the output from the opt.curve_fit command.
Defining The Function
When the opt.curve_fit command calls the function to find the best values for the coefficients, it calls it with an independent argument for each of the coefficients. That is to say, if the command is looking to solve a function for a straight line which has two coefficients - a slope and an intercept - the opt.curve_fit command will call the function with three inputs - the independent values, a guess for the slope, and a guess for the intercept. This is different from the paradigm we have used so far with a single list containing all the coefficients. In order to make the code more modular, we will use *args to take the coefficients in the function call and store them all in a single list. For example, to define a function for a straight line model using non-linear regression, the function call would be:
def yfun(x, *coefs):
return coefs[0] * x**1 + coefs[1] * x**0
Now the function can be called two different ways: either with three inputs or with two inputs where the first is the value or set of values for the independent variable and the second is an unpacked list or tuple with the items in it. That is to say, the following will calculate the \(y\) values for a line with the \(x\) values being a linearly spaced array of 5 values between 0 and 2 given a slope of 10 and an intercept of 8:
In [1]: yfun(np.linspace(0, 2, 5), 10, 8)
Out[1]: array([ 8., 13., 18., 23., 28.])
In [2]: yfun(np.linspace(0, 2, 5), *[10, 8])
Out[2]: array([ 8., 13., 18., 23., 28.])
The * before the list is key to unpack it into multiple parts; those multiple parts are then packed into the *arg.
Setting the Initial Conditions
Unlike the general linear regression method, which will find the best coefficients for a linear fit without needing any initial guess, the nonlinear regression method requires a good initial guess. Bad initial guesses can lead to errors or incorrect / non-optimal solutions. Note in the example code below that the initial guess gives 0.6 for the slope and 0.1 for the intercept. While these numbers are quite far from the optimized values of 0.0034 for the slope and 0.00055 for the intercept, the optimization routine is still able to find the correct value. That is not always the case - try to find an initial guess close to the actual answer. Sometimes the coefficients have names that may help - if some constant is called a_min you should perhaps guess the minimum a value in your data set.
Understanding the Output
The opt.curve_fit command returns two items in a tuple: the parameters themselves and some statistical information. Since you only want the first of these, it makes sense to put a [0] at the end of the command to just grab the parameter values. Remember that you will still need to unpack the list of parameters when you call your function.
Example for a Straight Line
1 # %% Import modules
2 import scipy.optimize as opt
3 from fitting_common import *
4
5 # %% Load and manipulate data
6 x, y, xmodel = get_beam_data()
7
8 # %% Perform calculations
9 def yfun(x, *coefs):
10 return coefs[0] * x ** 1 + coefs[1] * x ** 0
11
12 popt = opt.curve_fit(yfun, x, y, [0.6, 0.1])[0]
13 print(popt)
14
15
16
17
18
19
20
21 # %% Generate estimates and model
22 yhat = yfun(x, *popt)
23 ymodel = yfun(xmodel, *popt)
24
25 # %% Calculate statistics
26 calc_stats(y, yhat, 1)
27
28 # %% Generate plots
29 make_plot(x, y, yhat, xmodel, ymodel)
Example changes for different models
Polynomial
For the polynomial fitting model, really the only thing that would change would be the order of the fit and thus the value of \(n\) on line 23 of that code.
General Linear
For the general linear fit, the two places things will change will be in the function definition on line 10 and in the creation of the block matrix on line 14; the changes will mirror each other. For the straight line model - formally:
the code is
8 # %% Perform calculations
9 def yfun(xe, coefs):
10 return coefs[0] * xe**1 + coefs[1] * xe**0
11 # Reshape data for block matrices
12 xv = np.reshape(x, (-1, 1))
13 yv = np.reshape(y, (-1, 1))
14 phi_mat = np.block([[xv**1, xv**0]])
15 pvec = np.linalg.lstsq(phi_mat, yv, rcond=None)[0]
16 print(pvec)
where pvec[0][0] will be the slope \(m\) and pvec[1][0] will be the intercept \(b\). If the model were changed to, say,
which, formally, is
then the code would change to:
8 # %% Perform calculations
9 def yfun(xe, coefs):
10 return coefs[0] * np.cos(xe) + coefs[1] * np.sin(xe) + coefs[2] * xe**0
11 # Reshape data for block matrices
12 xv = np.reshape(x, (-1, 1))
13 yv = np.reshape(y, (-1, 1))
14 phi_mat = np.block([[np.cos(xv), np.sin(xv), xv**0]])
15 pvec = np.linalg.lstsq(phi_mat, yv, rcond=None)[0]
16 print(pvec)
and pvec will be a 2D array with three rows.
Nonlinear
For the nonlinear fit, the two places things will change will be in the function definition on line 10 and in the initial parameter value guess on line 12; there need to be as many values in the initial parameter list as there are parameters. For the straight line model - formally:
the code is
8 # %% Perform calculations
9 def yfun(x, *coefs):
10 return coefs[0] * x**1 + coefs[1] * x**0
11
12 popt = opt.curve_fit(yfun, x, y, [0.6, 0.1])[0]
13 print(popt)
where popt[0] will be the slope \(m\) and popt[1] will be the intercept \(b\). If the model were changed to, say,
which, formally, is
then the code would change to:
8 # %% Perform calculations
9 def yfun(x, *coefs):
10 return coefs[0] * np.cos(coefs[1]*x + coefs[2]) + coefs[3] * x**0
11
12 popt = opt.curve_fit(yfun, x, y, [.01, 1, np.pi/4, .0125])[0]
13 print(popt)
where [.01, 1, np.pi/4, .0125] represents initial guesses for the magnitude a, frequency b, phase c, and average value of the function. The popt variable would have 4 values.
