In Pandas, how to delete rows from a Data Frame based on another Data Frame?
I have 2 Data Frames, one named USERS and another named EXCLUDE. Both of them have a field named "email".
Basically, I want to remove every row in USERS that has an email contained in EXCLUDE.
How can I do it?
Solution 1:
You can use boolean indexing and condition with isin, inverting boolean Series is by ~:
import pandas as pd
USERS = pd.DataFrame({'email':['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']})
print (USERS)
email
0 [email protected]
1 [email protected]
2 [email protected]
3 [email protected]
4 [email protected]
EXCLUDE = pd.DataFrame({'email':['[email protected]','[email protected]']})
print (EXCLUDE)
email
0 [email protected]
1 [email protected]
print (USERS.email.isin(EXCLUDE.email))
0 True
1 False
2 False
3 False
4 True
Name: email, dtype: bool
print (~USERS.email.isin(EXCLUDE.email))
0 False
1 True
2 True
3 True
4 False
Name: email, dtype: bool
print (USERS[~USERS.email.isin(EXCLUDE.email)])
email
1 [email protected]
2 [email protected]
3 [email protected]
Another solution with merge:
df = pd.merge(USERS, EXCLUDE, how='outer', indicator=True)
print (df)
email _merge
0 [email protected] both
1 [email protected] left_only
2 [email protected] left_only
3 [email protected] left_only
4 [email protected] both
print (df.loc[df._merge == 'left_only', ['email']])
email
1 [email protected]
2 [email protected]
3 [email protected]
Solution 2:
Just to expand jezrael's answer, the same approach could be used in order to filter rows based on multiple columns.
USERS = pd.DataFrame({"email": ["[email protected]", "[email protected]", "[email protected]",
"[email protected]", "[email protected]"],
"name": ["a", "s", "d",
"f", "g"],
"nutrient_of_choice": ["pizza", "corn", "bread",
"coffee", "sausage"]})
print(USERS)
email name nutrient_of_choice
0 [email protected] a pizza
1 [email protected] s corn
2 [email protected] d bread
3 [email protected] f coffee
4 [email protected] g sausage
EXCLUDE = pd.DataFrame({"email":["[email protected]", "[email protected]"],
"name": ["a", "f"]})
print(EXCLUDE)
email name
0 [email protected] a
1 [email protected] f
Now, suppose we would like to filter only rows with matching names and emails:
USERS = pd.merge(USERS, EXCLUDE, on=["email", "name"], how="outer", indicator=True)
print(USERS)
email name nutrient_of_choice _merge
0 [email protected] a pizza left_only
1 [email protected] s corn left_only
2 [email protected] d bread left_only
3 [email protected] f coffee both
4 [email protected] g sausage left_only
5 [email protected] a NaN right_only
USERS = USERS.loc[USERS["_merge"] == "left_only"].drop("_merge", axis=1)
print(USERS)
email name nutrient_of_choice
0 [email protected] a pizza
1 [email protected] s corn
2 [email protected] d bread
4 [email protected] g sausage
Solution 3:
You can also use inner join, take the indices or rows in USERS, that has email EXCLUDE, and then drop the them from the USERS. Following I use the @jezrael example to show this:
import pandas as pd
USERS = pd.DataFrame({'email': ['[email protected]',
'[email protected]',
'[email protected]',
'[email protected]',
'[email protected]']})
EXCLUDE = pd.DataFrame({'email':['[email protected]',
'[email protected]']})
# rows in USERS and EXCLUDE with the same email
duplicates = pd.merge(USERS, EXCLUDE, how='inner',
left_on=['email'], right_on=['email'],
left_index=True)
# drop the indices from USERS
USERS = USERS.drop(duplicates.index)
This return:
USERS
email
2 [email protected]
3 [email protected]
4 [email protected]