Large Dataset Polynomial Fitting Using Numpy

I'm trying to fit a second order polynomial to raw data and output the results using Matplotlib. There are about a million points in the data set that I'm trying to fit. It is supposed to be simple, with many examples available around the web. However for some reason I cannot get it right.

I get the following warning message:

RankWarning: Polyfit may be poorly conditioned

This is my output:

This is output using Excel:

See below for my code. What am I missing??

xData = df['X']
yData = df['Y']
xTitle = 'X'
yTitle = 'Y'
title = ''
minX = 100
maxX = 300
minY = 500
maxY = 2200

title_font = {'fontname':'Arial', 'size':'30', 'color':'black', 'weight':'normal',
              'verticalalignment':'bottom'} # Bottom vertical alignment for more space
axis_font = {'fontname':'Arial', 'size':'18'}

#Poly fit

# calculate polynomial
z = np.polyfit(xData, yData, 2)
f = np.poly1d(z)
print(f)

# calculate new x's and y's
x_new = xData
y_new = f(x_new)   

#Plot
plt.scatter(xData, yData,c='#002776',edgecolors='none')
plt.plot(x_new,y_new,c='#C60C30')

plt.ylim([minY,maxY])
plt.xlim([minX,maxX])

plt.xlabel(xTitle,**axis_font)
plt.ylabel(yTitle,**axis_font)
plt.title(title,**title_font)

plt.show()      

Solution 1:

The array to plot must be sorted. Here is a comparisson between plotting a sorted and an unsorted array. The plot in the unsorted case looks completely distorted, however, the fitted function is of course the same.

        2
-3.496 x + 2.18 x + 17.26

import matplotlib.pyplot as plt
import numpy as np; np.random.seed(0)

x = (np.random.normal(size=300)+1)
fo = lambda x: -3*x**2+ 1.*x +20. 
f = lambda x: fo(x) + (np.random.normal(size=len(x))-0.5)*4
y = f(x)

fig, (ax, ax2) = plt.subplots(1,2, figsize=(6,3))
ax.scatter(x,y)
ax2.scatter(x,y)

def fit(ax, x,y, sort=True):
    z = np.polyfit(x, y, 2)
    fit = np.poly1d(z)
    print(fit)
    ax.set_title("unsorted")
    if sort:
        x = np.sort(x)
        ax.set_title("sorted")
    ax.plot(x, fo(x), label="original func", color="k", alpha=0.6)
    ax.plot(x, fit(x), label="fit func", color="C3", alpha=1, lw=2.5  )  
    ax.legend()


fit(ax, x,y, sort=False)

fit(ax2, x,y, sort=True) 


plt.show()