how to calculate error part in of fitted line of polyfit to data?

I am confused about the error part of polyfit. I have the following code:

def polyfit(df,columns, degree):
    coef=[]
    error=[]
    x = np.array(list(range(0,df.shape[0])))
    for skill in columns:
        y=df[skill]
        y=pd.to_numeric(y)
        coeffs = numpy.polyfit(x, y, degree)

        # Polynomial Coefficients
        coef.append(coeffs.tolist()[0])

        # r-squared
        p = numpy.poly1d(coeffs)
        # fit values, and mean
        yhat = p(x)                         # or [p(z) for z in x]
        ybar = numpy.sum(y)/len(y)          # or sum(y)/len(y)
        ssreg = numpy.sum((yhat-ybar)**2)   # or sum([ (yihat - ybar)**2 for yihat in yhat])
        sstot = numpy.sum((y - yhat)**2)    # or sum([ (yi - ybar)**2 for yi in y])
        error.append(ssreg / sstot)
        
    results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':sstot})
    return results

where a sample of df is:

new_list    administrative coordination administrative law  administrative support
0   0.0465116   0.0232558   0.0581395
1   0.0714286   0   0.0285714
2   0.0210526   0   0.0421053
3   0.0288462   0.00961538  0.0961538
4   0.0714286   0.0238095   0.107143
5   0.00952381  0   0.0666667
6   0.0285714   0.00952381  0.0666667
7   0.0428571   0   0.0428571
8   0.111111    0.0277778   0.138889
9   0   0.0136986   0.0273973

the result is as below:

polyfit(df,['administrative coordination',  'administrative law',   'administrative support'], 1)

skills  coef    error   error2
0   administrative coordination -0.000573   0.002681    0.011538
1   administrative law  0.000511    0.020165    0.011538
2   administrative support  0.002245    0.036025    0.011538

But why error2 is the same for all columns? where i am making mistake about calculating error part? I wanna find columns with minimum errors. By error I mean minimum distances of fitted line to data points.


Your variable sstot is a scalar and will be reset for each iteration of your loop. This means that when you run the following line:

results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':sstot}) 

The column error2 will be set to scalar value of sstot in the most recent iteration of your for loop, which is why error2 has all the same value.

I am guessing that you meant to keep track of sstot for each skill, so you can create a list called error2, then set the column error2equal to this list (like the lists you created forcoefanderror`). For example:

def polyfit(df,columns, degree):
    coef=[]
    error=[]
    error2=[]
    x = np.array(list(range(0,df.shape[0])))
    for skill in columns:
        y=df[skill]
        y=pd.to_numeric(y)
        coeffs = np.polyfit(x, y, degree)

        # Polynomial Coefficients
        coef.append(coeffs.tolist()[0])

        # r-squared
        p = np.poly1d(coeffs)

        # fit values, and mean
        yhat = p(x)                         # or [p(z) for z in x]
        ybar = np.sum(y)/len(y)          # or sum(y)/len(y)
        ssreg = np.sum((yhat-ybar)**2)   # or sum([ (yihat - ybar)**2 for yihat in yhat])
        sstot = np.sum((y - yhat)**2)    # or sum([ (yi - ybar)**2 for yi in y])
        error.append(ssreg / sstot)
        error2.append(sstot)

    results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':error2})
    return results

Result using your sample df:

>>> polyfit(df,['administrative coordination',  'administrative law',   'administrative support'], 1)
                        skills      coef     error    error2
0  administrative coordination -0.000573  0.002681  0.010100
1           administrative law  0.000511  0.020165  0.001069
2       administrative support  0.002245  0.036025  0.011538