Filling missing values using forward and backward fill in pandas dataframe (ffill and bfill)
Beginner with panda dataframes. I have this data set below with missing values for column A and B (Test.csv):
DateTime A B
01-01-2017 03:27
01-01-2017 03:28
01-01-2017 03:29 0.18127718 -0.178835737
01-01-2017 03:30 0.186923018 -0.183260853
01-01-2017 03:31
01-01-2017 03:32
01-01-2017 03:33 0.18127718 -0.178835737
I can use this code to fill in values using forward propagation, but this only fills in for 03:31 and 03:32, and not 03:27 and 03:28.
import pandas as pd
import numpy as np
df = pd.read_csv('test.csv', index_col = 0)
data = df.fillna(method='ffill')
ndata = data.to_csv('test1.csv')
results in:
DateTime A B
01-01-2017 03:27
01-01-2017 03:28
01-01-2017 03:29 0.18127718 -0.178835737
01-01-2017 03:30 0.186923018 -0.183260853
01-01-2017 03:31 0.186923018 -0.183260853
01-01-2017 03:32 0.186923018 -0.183260853
01-01-2017 03:33 0.18127718 -0.178835737
How could I include the 'Bfill' to fill in the missing values for 03:27 and 03:28 using the backfil?
Solution 1:
You can use ffill
and bfill
if need replace NaN
values forward and backward filling:
print (df)
A B
DateTime
01-01-2017 03:27 NaN NaN
01-01-2017 03:28 NaN NaN
01-01-2017 03:29 0.181277 -0.178836
01-01-2017 03:30 0.186923 -0.183261
01-01-2017 03:31 NaN NaN
01-01-2017 03:32 NaN NaN
01-01-2017 03:33 0.181277 -0.178836
data = df.ffill().bfill()
print (data)
A B
DateTime
01-01-2017 03:27 0.181277 -0.178836
01-01-2017 03:28 0.181277 -0.178836
01-01-2017 03:29 0.181277 -0.178836
01-01-2017 03:30 0.186923 -0.183261
01-01-2017 03:31 0.186923 -0.183261
01-01-2017 03:32 0.186923 -0.183261
01-01-2017 03:33 0.181277 -0.178836
Which is same as the function fillna
with parameters:
data = df.fillna(method='ffill').fillna(method='bfill')