Python pandas: how to remove nan and -inf values
Use pd.DataFrame.isin
and check for rows that have any with pd.DataFrame.any
. Finally, use the boolean array to slice the dataframe.
df[~df.isin([np.nan, np.inf, -np.inf]).any(1)]
time X Y X_t0 X_tp0 X_t1 X_tp1 X_t2 X_tp2
4 0.037389 3 10 3 0.333333 2.0 0.500000 1.0 1.000000
5 0.037393 4 10 4 0.250000 3.0 0.333333 2.0 0.500000
1030308 9.962213 256 268 256 0.000000 256.0 0.003906 255.0 0.003922
You can replace inf
and -inf
with NaN
, and then select non-null rows.
df[df.replace([np.inf, -np.inf], np.nan).notnull().all(axis=1)] # .astype(np.float64) ?
or
df.replace([np.inf, -np.inf], np.nan).dropna(axis=1)
Check the type of your columns returns to make sure they are all as expected (e.g. np.float32/64) via df.info()
.
df.replace([np.inf, -np.inf], np.nan)
df.dropna(inplace=True)