Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Session returned an error

Highlighted

Spark Session returned an error

Explorer

We are trying to run a spark program using NiFi. [This is the basic sample][1] we tried to follow.

We have configured Apache-Livy server in

 

127.0.0.1:8998

 

ExecutiveSparkInteractive processor is used to run sample Spark code.

 

 

val gdpDF = spark.read.json("gdp.json")
val gdpRDD = gdpDF.rdd
gdpRDD.count()

 



LivyController is configured for 127.0.0.1 port 3998 and Session Type : spark.

 

When we run the processor we get following error :

 

Spark Session returned an error, sending the output JSON object as the flow file content to failure (after penalizing)

 

We just want to output the line count in JSON file. How to redirect it to flowfile?

[1]: https://medium.com/@evanescence1106/in-this-article-we-will-use-apache-nifi-to-schedule-batch-jobs-i...

Don't have an account?
Coming from Hortonworks? Activate your account here