Spark scala column level mismatches from 2 dataframes

Maybe this will help:

  val spark = SparkSession.builder.appName("Test").master("local[*]").getOrCreate();

  import spark.implicits._

  var df1 = Seq((1, "1", "6"), (2, "10", "8"), (3, "6", "4")).toDF("id", "value1", "value2")
  var df2 = Seq((1, "1", "6"), (2, "5", "4"), (3, "3", "1")).toDF("id", "value1", "value2")

  df1.columns.foreach(column => {
    df1 = df1.withColumn(column, df1.col(column).cast(IntegerType))
  })
  df2.columns.foreach(column => {
    df2 = df2.withColumn(column, df2.col(column).cast(IntegerType))
  })

  df1 = df1.withColumnRenamed("id", "df1_id")
  df2 = df2.withColumnRenamed("id", "df2_id")

  df1.show()
  df2.show()

so till now you have two dataframes with value_x,value_y,value_z and going on ...

df1:

+------+------+------+
|df1_id|value1|value2|
+------+------+------+
|     1|     1|     6|
|     2|    10|     8|
|     3|     6|     4|
+------+------+------+

df2:

+------+------+------+
|df2_id|value1|value2|
+------+------+------+
|     1|     1|     6|
|     2|     5|     4|
|     3|     3|     1|
+------+------+------+

Now we are gonna join them base on id:

  var df3 = df1.alias("df1").join(df2.alias("df2"), $"df1.df1_id" === $"df2.df2_id")

and last, we gonna take all columns on df1/df2 (* Its important that they will have the same columns) - without the id, and create a new column of the diff:

  df1.columns.tail.foreach(col => {
    val new_col_name = s"${col}-diff"
    val df_a_col = s"df1.${col}"
    val df_b_col = s"df2.${col}"
    df3 = df3.withColumn(new_col_name, df3.col(df_a_col) - df3.col(df_b_col))
  })

  df3.show()

Result:

+------+------+------+------+------+------+-----------+-----------+
|df1_id|value1|value2|df2_id|value1|value2|value1-diff|value2-diff|
+------+------+------+------+------+------+-----------+-----------+
|     1|     1|     6|     1|     1|     6|          0|          0|
|     2|    10|     8|     2|     5|     4|          5|          4|
|     3|     6|     4|     3|     3|     1|          3|          3|
+------+------+------+------+------+------+-----------+-----------+

This is the result, and it`s dynamic so you can add valueX you want.