NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

[#1]

CODE : executesparkinteractive-code.txt

ERROR :

2018-06-06 15:32:13,566 ERROR [Timer-Driven Process Thread-5] o.a.n.p.livy.ExecuteSparkInteractive ExecuteSparkInteractive[id=aeb74038-5333-13d2-0000-00001ea7e32e] ExecuteSparkInteractive[i
d=aeb74038-5333-13d2-0000-00001ea7e32e] failed to process session due to java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request: {}
java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.readJSONObjectFromUrlPOST(ExecuteSparkInteractive.java:282)
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.submitAndHandleJob(ExecuteSparkInteractive.java:234)
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.onTrigger(ExecuteSparkInteractive.java:197)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[#2]

Problem was in "sqlContext.registerDataFrameAsTable"

What investigate similar (HTTP) errors ?

[#3]

@Dmitro Vasilenko I would recommend you review Livy logs under /var/log/livy2 to understand why the application submission is failing.

HTH

[#4]

Hi! one more question, how control count "livy-session-"" ?

[#5]

Hi ,

Any solution for this. I am also facing same issue.

[#6]

Hi! Try One of this changes:

Zeppelin	NIFI ExecuteSparkInteractive
preds = spark.sql(‘select * from sandbox.CHURN_PRP_D_PREDS_S ’)	preds = spark.sql("select * from andbox.CHURN_PRP_D_PREDS_S")
dir = ‘ /user/CHURN_PRP_D’	dir = "hdfs:/user/CHURN_PRP_D"
date_df = spark.sql("select to_date('{}') as score_date".format(date_calc))	ttt = spark.createDataFrame([(date_calc,)], ["t"]) date_df = ttt.select(to_date(ttt.t))
sqlContext.registerDataFrameAsTable(train, "train")	train.registerTempTable("train")

[#7]

Hi,

I am trying to execute the below statements as part of code values in processor :

import org.apache.spark.sql.hive.HiveContext
val hiveContext = new HiveContext(sc)
import hiveContext._
hiveContext.sql("insert into cpv.del values('from livy')")

After looking recent comments from @Dmitro Vasilenko @, I have changed code to

spark.sql("insert into dbname.tablename values('from livy')")

Still Getting bad request error. Please look into attachements for configuration parameters. tempsnip.png tempsnipn.png

Looking for quick response, Thanks

[#8]

Hi @Dmitro Vasilenko,

did u get a chance to look this. @nifi @ExecuteSparkInteractive @spark @Timothy Spann

spark.sql("insert into dbname.tablename values('from livy')") is not working

spark.sql("insert into dbname.tablename select * from dbname.tablename") is working

For more information, please look into my previous posts on same page/tasl/requiement.

@nifi

@executesparkintactive