how to calculate error part in of fitted line of polyfit to data?
I am confused about the error part of polyfit. I have the following code:
def polyfit(df,columns, degree):
coef=[]
error=[]
x = np.array(list(range(0,df.shape[0])))
for skill in columns:
y=df[skill]
y=pd.to_numeric(y)
coeffs = numpy.polyfit(x, y, degree)
# Polynomial Coefficients
coef.append(coeffs.tolist()[0])
# r-squared
p = numpy.poly1d(coeffs)
# fit values, and mean
yhat = p(x) # or [p(z) for z in x]
ybar = numpy.sum(y)/len(y) # or sum(y)/len(y)
ssreg = numpy.sum((yhat-ybar)**2) # or sum([ (yihat - ybar)**2 for yihat in yhat])
sstot = numpy.sum((y - yhat)**2) # or sum([ (yi - ybar)**2 for yi in y])
error.append(ssreg / sstot)
results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':sstot})
return results
where a sample of df is:
new_list administrative coordination administrative law administrative support
0 0.0465116 0.0232558 0.0581395
1 0.0714286 0 0.0285714
2 0.0210526 0 0.0421053
3 0.0288462 0.00961538 0.0961538
4 0.0714286 0.0238095 0.107143
5 0.00952381 0 0.0666667
6 0.0285714 0.00952381 0.0666667
7 0.0428571 0 0.0428571
8 0.111111 0.0277778 0.138889
9 0 0.0136986 0.0273973
the result is as below:
polyfit(df,['administrative coordination', 'administrative law', 'administrative support'], 1)
skills coef error error2
0 administrative coordination -0.000573 0.002681 0.011538
1 administrative law 0.000511 0.020165 0.011538
2 administrative support 0.002245 0.036025 0.011538
But why error2 is the same for all columns? where i am making mistake about calculating error part? I wanna find columns with minimum errors. By error I mean minimum distances of fitted line to data points.
Your variable sstot
is a scalar and will be reset for each iteration of your loop. This means that when you run the following line:
results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':sstot})
The column error2
will be set to scalar value of sstot in the most recent iteration of your for loop, which is why error2 has all the same value.
I am guessing that you meant to keep track of sstot for each skill
, so you can create a list called error2, then set the column
error2equal to this list (like the lists you created for
coefand
error`). For example:
def polyfit(df,columns, degree):
coef=[]
error=[]
error2=[]
x = np.array(list(range(0,df.shape[0])))
for skill in columns:
y=df[skill]
y=pd.to_numeric(y)
coeffs = np.polyfit(x, y, degree)
# Polynomial Coefficients
coef.append(coeffs.tolist()[0])
# r-squared
p = np.poly1d(coeffs)
# fit values, and mean
yhat = p(x) # or [p(z) for z in x]
ybar = np.sum(y)/len(y) # or sum(y)/len(y)
ssreg = np.sum((yhat-ybar)**2) # or sum([ (yihat - ybar)**2 for yihat in yhat])
sstot = np.sum((y - yhat)**2) # or sum([ (yi - ybar)**2 for yi in y])
error.append(ssreg / sstot)
error2.append(sstot)
results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':error2})
return results
Result using your sample df
:
>>> polyfit(df,['administrative coordination', 'administrative law', 'administrative support'], 1)
skills coef error error2
0 administrative coordination -0.000573 0.002681 0.010100
1 administrative law 0.000511 0.020165 0.001069
2 administrative support 0.002245 0.036025 0.011538