Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

avatar
Expert Contributor
CODE : executesparkinteractive-code.txt

ERROR :

2018-06-06 15:32:13,566 ERROR [Timer-Driven Process Thread-5] o.a.n.p.livy.ExecuteSparkInteractive ExecuteSparkInteractive[id=aeb74038-5333-13d2-0000-00001ea7e32e] ExecuteSparkInteractive[i
d=aeb74038-5333-13d2-0000-00001ea7e32e] failed to process session due to java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request: {}
java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.readJSONObjectFromUrlPOST(ExecuteSparkInteractive.java:282)
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.submitAndHandleJob(ExecuteSparkInteractive.java:234)
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.onTrigger(ExecuteSparkInteractive.java:197)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

76589-executesparkinteractive.png

7 REPLIES 7

avatar
Expert Contributor

Problem was in "sqlContext.registerDataFrameAsTable"

What investigate similar (HTTP) errors ?

avatar

@Dmitro Vasilenko I would recommend you review Livy logs under /var/log/livy2 to understand why the application submission is failing.

HTH

avatar
Expert Contributor

Hi! one more question, how control count "livy-session-"" ?

avatar
Explorer

Hi ,

Any solution for this. I am also facing same issue.

avatar
Expert Contributor

Hi! Try One of this changes:

Zeppelin NIFI ExecuteSparkInteractive
preds = spark.sql(‘select * from sandbox.CHURN_PRP_D_PREDS_S ’) preds = spark.sql("select * from andbox.CHURN_PRP_D_PREDS_S")
dir = ‘ /user/CHURN_PRP_D’ dir = "hdfs:/user/CHURN_PRP_D"
date_df = spark.sql("select to_date('{}') as score_date".format(date_calc)) ttt = spark.createDataFrame([(date_calc,)], ["t"])

date_df = ttt.select(to_date(ttt.t))

sqlContext.registerDataFrameAsTable(train, "train") train.registerTempTable("train")

avatar
Explorer

Hi,

I am trying to execute the below statements as part of code values in processor :

import org.apache.spark.sql.hive.HiveContext
val hiveContext = new HiveContext(sc)
import hiveContext._
hiveContext.sql("insert into cpv.del values('from livy')")

After looking recent comments from @Dmitro Vasilenko @, I have changed code to

spark.sql("insert into dbname.tablename values('from livy')")

Still Getting bad request error. Please look into attachements for configuration parameters. tempsnip.png tempsnipn.png

Looking for quick response, Thanks

avatar
Explorer

Hi @Dmitro Vasilenko,

did u get a chance to look this. @nifi @ExecuteSparkInteractive @spark @Timothy Spann

spark.sql("insert into dbname.tablename values('from livy')") is not working

spark.sql("insert into dbname.tablename select * from dbname.tablename") is working

For more information, please look into my previous posts on same page/tasl/requiement.

@nifi

@executesparkintactive