Spark doesn't recognize the column name in SQL query while can output it to a dataset

Solution 1:

From the exception, it seems this is related to Postgres not Spark. If you look at the error message you got, the column name is folded to lowercase accepttimestamp whereas in your query the T is in uppercase acceptTimestamp.

To make the column name case-sensitive for Postgres, you need to use double-quotes. Try this:

val query = s"""SELECT * FROM my_table_joined 
    WHERE   timestamp > '2022-01-23' 
    and     writetime is not null 
    and     "acceptTimestamp" is not null"""

Spark doesn't recognize the column name in SQL query while can output it to a dataset

Solution 1:

Related

Recent Posts