Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Highlighted

NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Contributor
CODE : executesparkinteractive-code.txt

ERROR :

2018-06-06 15:32:13,566 ERROR [Timer-Driven Process Thread-5] o.a.n.p.livy.ExecuteSparkInteractive ExecuteSparkInteractive[id=aeb74038-5333-13d2-0000-00001ea7e32e] ExecuteSparkInteractive[i
d=aeb74038-5333-13d2-0000-00001ea7e32e] failed to process session due to java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request: {}
java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.readJSONObjectFromUrlPOST(ExecuteSparkInteractive.java:282)
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.submitAndHandleJob(ExecuteSparkInteractive.java:234)
        at org.apache.nifi.processors.livy.ExecuteSparkInteractive.onTrigger(ExecuteSparkInteractive.java:197)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

76589-executesparkinteractive.png

7 REPLIES 7
Highlighted

Re: NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Contributor

Problem was in "sqlContext.registerDataFrameAsTable"

What investigate similar (HTTP) errors ?

Highlighted

Re: NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

@Dmitro Vasilenko I would recommend you review Livy logs under /var/log/livy2 to understand why the application submission is failing.

HTH

Highlighted

Re: NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Contributor

Hi! one more question, how control count "livy-session-"" ?

Highlighted

Re: NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Explorer

Hi ,

Any solution for this. I am also facing same issue.

Re: NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Contributor

Hi! Try One of this changes:

Zeppelin NIFI ExecuteSparkInteractive
preds = spark.sql(‘select * from sandbox.CHURN_PRP_D_PREDS_S ’) preds = spark.sql("select * from andbox.CHURN_PRP_D_PREDS_S")
dir = ‘ /user/CHURN_PRP_D’ dir = "hdfs:/user/CHURN_PRP_D"
date_df = spark.sql("select to_date('{}') as score_date".format(date_calc)) ttt = spark.createDataFrame([(date_calc,)], ["t"])

date_df = ttt.select(to_date(ttt.t))

sqlContext.registerDataFrameAsTable(train, "train") train.registerTempTable("train")
Highlighted

Re: NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Explorer

Hi,

I am trying to execute the below statements as part of code values in processor :

import org.apache.spark.sql.hive.HiveContext
val hiveContext = new HiveContext(sc)
import hiveContext._
hiveContext.sql("insert into cpv.del values('from livy')")

After looking recent comments from @Dmitro Vasilenko @, I have changed code to

spark.sql("insert into dbname.tablename values('from livy')")

Still Getting bad request error. Please look into attachements for configuration parameters. tempsnip.png tempsnipn.png

Looking for quick response, Thanks

Highlighted

Re: NIFI ExecuteSparkInteractive Error HTTP error code : 400 : Bad Request

Explorer

Hi @Dmitro Vasilenko,

did u get a chance to look this. @nifi @ExecuteSparkInteractive @spark @Timothy Spann

spark.sql("insert into dbname.tablename values('from livy')") is not working

spark.sql("insert into dbname.tablename select * from dbname.tablename") is working

For more information, please look into my previous posts on same page/tasl/requiement.

@nifi

@executesparkintactive

Don't have an account?
Coming from Hortonworks? Activate your account here