What: Executing Scala Apache Spark Code in JARS from Apache NiFi Why: You don't want all of your scala code in a continuous block like Apache Zeppelin Tools: Apache NiFi - Apache Livy - Apache Spark - Scala Flows: Option 1: Inline Scala Code Apache Zeppelin Running the Same Scala Job (have to add the jar to interpreter for Spark and restart) Grafana Charts of Apache NiFi Run Log Search Helps You Find Errors Run Code For Your Spark Class Setting up Your ExecuteSparkInteractive Processor Setting Up Your Spark Service for Scala Tracking the Job in Livy UI Tracking the Job in Spark UI I was looking at doing this: Pull code from Git and put into a NiFi attribute, run directly. For bigger projects, you will have many classes and dependencies that may require a full IDE and SBT build cycle. Once I build a Scala jar I want to run against that. Example Code package com.dataflowdeveloper.example import org.apache.spark.sql.SparkSession class Example () { def run( spark: SparkSession) { try { println("Started") val shdf = spark.read.json("hdfs://princeton0.field.hortonworks.com:8020/smartPlugRaw") shdf.printSchema() shdf.createOrReplaceTempView("smartplug") val stuffdf = spark.sql("SELECT * FROM smartplug") stuffdf.count() println("Complete.") } catch { case e: Exception => e.printStackTrace(); } } } =--- Run that with import com.dataflowdeveloper.example.Example println("Before run") val job = new Example() job.run(spark) println("After run") === after run {"text\/plain":"After run"} Import Tip You need to put your Jar in Session.jars on the Session control and on the same directory on hdfs. So I did /opt/demo/example.jar in Linux and hdfs:// /opt/demo/example.jar. Make sure Livy and NiFi have read permissions on those. Github: https://github.com/tspannhw/livysparkjob Github Release: https://github.com/tspannhw/livysparkjob/releases/tag/v1.1 Apache NiFi Flow Example spark-it-up-scala.xml ... View more