Python/pyspark data frame rearrange columns
I have a data frame in python/pyspark with columns id
time
city
zip
and so on......
Now I added a new column name
to this data frame.
Now I have to arrange the columns in such a way that the name
column comes after id
I have done like below
change_cols = ['id', 'name']
cols = ([col for col in change_cols if col in df]
+ [col for col in df if col not in change_cols])
df = df[cols]
I am getting this error
pyspark.sql.utils.AnalysisException: u"Reference 'id' is ambiguous, could be: id#609, id#1224.;"
Why is this error occuring. How can I rectify this.
You can use select
to change the order of the columns:
df.select("id","name","time","city")
If you're working with a large number of columns:
df.select(sorted(df.columns))