Support Questions
Find answers, ask questions, and share your expertise

Spark Session returned an error

Explorer

We are trying to run a spark program using NiFi. [This is the basic sample][1] we tried to follow.

We have configured Apache-Livy server in

 

127.0.0.1:8998

 

ExecutiveSparkInteractive processor is used to run sample Spark code.

 

 

val gdpDF = spark.read.json("gdp.json")
val gdpRDD = gdpDF.rdd
gdpRDD.count()

 



LivyController is configured for 127.0.0.1 port 3998 and Session Type : spark.

 

When we run the processor we get following error :

 

Spark Session returned an error, sending the output JSON object as the flow file content to failure (after penalizing)

 

We just want to output the line count in JSON file. How to redirect it to flowfile?

[1]: https://medium.com/@evanescence1106/in-this-article-we-will-use-apache-nifi-to-schedule-batch-jobs-i...

1 REPLY 1

Super Collaborator

I have seen several people struggling with this example. I recommend following this example from the Cloudera Community (especially note part 2).
https://community.cloudera.com/t5/Community-Articles/HDF-3-1-Executing-Apache-Spark-via-ExecuteSpark...


The key points I would be concerned with:

1.Does your spark work in general
2. Does your livy work in general
3. Is the Spark sample code good

 


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
; ;