Jackson conflicts in Apache Spark when used with Azure Java SDK
Azure Synapse publishes the jars available at runtime here. I am currently using Apache Spark 3.1 runtime.
My project also relies on the 1.4.0 version of azure-eventgrid as a dependency (which pulls in azure-core). I am getting the following errors when the job is deployed on Synapse.
The job functions fine locally, but not when deployed on Synapse.
21/11/29 17:38:00 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.LinkageError: Package versions: jackson-annotations=2.10.0, jackson-core=2.10.0, jackson-databind=2.10.0, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.12.5, azure-core=1.19.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:73) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:81) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:58) at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.(JacksonAdapter.java:113) at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122) at com.azure.identity.implementation.IdentityClient.(IdentityClient.java:100) at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:139) at com.azure.identity.ManagedIdentityCredential.(ManagedIdentityCredential.java:70) at com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:129) at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:123)com.xxxxxxxxxxx.$anonfun$sendEvents$1$adapted(xxxxxxxGridSender.scala:25) at scala.collection.immutable.List.foreach(List.scala:392) at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.sendEvents(xxxxxx.scala:25) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at xxxxxx.xxxxxx.xxxxxx.xxxxxx.runner.xxxxxx.xxxxxx(xxxxxx.scala:82) at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx(xxxxxx.scala:61) at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.$anonfun$start$2(xxxxxx.scala:39) at scala.collection.TraversableLike$WithFilter.$anonfun$map$2(TraversableLike.scala:827) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:826) at xxxxxx.xxxxxx.xxxxxx.xxxxxx.x.xxxxx.xxxxxx.start(xxxxxx.scala:36) at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx$.main(xxxxxx.scala:29) at xxxxxx.xxxxxx.aiops.xxxxxx.xxxxxx.main(xxxxxx.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732) Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig; at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:176) at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:145) at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:127) at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218) at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:84) at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:70) ... 45 more
Synapse has its own jars available as part of its runtime. The project dependencies need to be compatible with the jars available at the runtime.
There are two parts here:
- Azure-core gets in Jackson dependency 2.12 series. Where as Apache Spark 3.1 is still on 2.10 series.
- Azure-core is already available on the classpath of synapse (1.16.0). So any azure lib that one pulls in (which along with it, gets azure-core as a dependency), needs to be compatible with azure-core 1.16.0
To fix (1), I added the following:
object DependencyOverrides {
/**
* We do not have any direct dependency on jackson. Spark relies on 2.10 series and Azure-core sdk has dependency on 2.12.
* In order to resolve conflicts, we explicitly provide the jackson dependency here to 2.10.5
*/
val jackson = Seq(
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.10.0",
"com.fasterxml.jackson.core" % "jackson-core" % "2.10.0",
"com.fasterxml.jackson.core" % "jackson-annotations" % "2.10.0",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.10.0",
"com.fasterxml.jackson.dataformat" % "jackson-dataformat-xml" % "2.10.0",
"com.fasterxml.jackson.datatype" % "jackson-datatype-jsr310" % "2.10.0",
)
val others = Seq(
"com.google.guava" % "guava" % "27.0-jre"
)
val all = jackson ++ others
}
And override the above dependency in SBT:
dependencyOverrides ++= DependencyOverrides.all
To fix (2), Additionally add related jars in others above:
val others = Seq(
"com.azure" % "azure-core" % "1.16.0",
"com.azure" % "azure-core-http-netty" % "1.6.2",
"com.google.guava" % "guava" % "27.0-jre"
)
In my case, adding azure-core was not just sufficient. Had to add, azure-core-http-netty and guava as well.