Pyspark: Convert column to lowercase
I want to convert the values inside a column to lowercase. Currently if I use the lower()
method, it complains that column objects are not callable. Since there's a function called lower()
in SQL, I assume there's a native Spark solution that doesn't involve UDFs, or writing any SQL.
Import lower
alongside col
:
from pyspark.sql.functions import lower, col
Combine them together using lower(col("bla"))
. In a complete query:
spark.table('bla').select(lower(col('bla')).alias('bla'))
which is equivalent to the SQL query
SELECT lower(bla) AS bla FROM bla
To keep the other columns, do
spark.table('foo').withColumn('bar', lower(col('bar')))
Needless to say, this approach is better than using a UDF because UDFs have to call out to Python (which is a slow operation, and Python itself is slow), and is more elegant than writing it in SQL.
You can use a combination of concat_ws and split
from pyspark.sql.functions import *
df.withColumn('arr_str', lower(concat_ws('::','arr'))).withColumn('arr', split('arr_str','::')).drop('arr_str')