Can I use another build system to build custom processors with NiFi?
My company uses SBT/Scala so my goal is to develop our custom NiFi processors with these rather than Maven/Java. The problem I'm having is I don't know how to generate the NAR files with SBT. NiFi has a nifi-nar-maven-plugin package plugin that it uses to package all the processor dependencies into a jar (nar). Is there anyway to build the NAR using SBT?
Options I've considered:
- Building a fat jar (ex: https://github.com/sbt/sbt-assembly). I see pom files in the NAR but are those actually used for anything or is it just meta data? I see classes like NarUnpacker, NarClassLoader so I'm wondering if there is something specific in the Nar format that would prevent Fat jars from working.
- Reverse engineering the plugin and create an SBT one. Could be a bit of work and then we'd risk getting out of sync with changes to the original plugin.
- Build a wrapper around the maven plugin for SBT. I don't think this will work however since the plugin's built off of maven libraries.
- Leaving just the NiFi sub project as a maven build and running it via the SBT parent project. (ex. http://www.scala-sbt.org/0.13/docs/Process.html)
- Give up and convince management to use maven.
Solution 1:
NARs are used to provided class-loader isolation in NiFi. Each NAR is an artifact that contains all of the JARs it needs, and they are only visible to that NAR. This prevents against conflicting libraries used by different NARs, so if one NAR uses Guava 16 and another NAR uses Guava 18, it won't cause a problem.
NARs get unpacked to nifi_home/work/nar. For example, looking at nifi-0.7.0/work/nar/extensions/nifi-ambari-nar-0.7.0.nar-unpacked/META-INF/bundled-dependencies/ it would show all the JARS that were included for the ambari NAR.
Each NAR can have a dependency on one other NAR, this is how processors can have a NAR dependency on controller services. In these cases, the NAR Maven plugin creates a special MANIFEST file in the NAR that specifies the id of a parent NAR, and that id is used by NiFi to perform the appropriate class loading.
The developer guide covers some of this: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
Personally I would recommend using Maven since the NAR plugin is the standard way NARs are produced. As you pointed out it is probably possible to create a similar plugin with SBT, but seems like a lot of effort. Using Maven doesn't mean you can't use Scala... This project offers a template for developing processors in Scala, but still uses Maven as the build tool:
https://github.com/jfrazee/nifi-processor-bundle-scala.g8