Created on 06-06-2018 12:38 PM - edited 08-17-2019 09:11 PM
CODE : executesparkinteractive-code.txt ERROR : 2018-06-06 15:32:13,566 ERROR [Timer-Driven Process Thread-5] o.a.n.p.livy.ExecuteSparkInteractive ExecuteSparkInteractive[id=aeb74038-5333-13d2-0000-00001ea7e32e] ExecuteSparkInteractive[i d=aeb74038-5333-13d2-0000-00001ea7e32e] failed to process session due to java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request: {} java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request at org.apache.nifi.processors.livy.ExecuteSparkInteractive.readJSONObjectFromUrlPOST(ExecuteSparkInteractive.java:282) at org.apache.nifi.processors.livy.ExecuteSparkInteractive.submitAndHandleJob(ExecuteSparkInteractive.java:234) at org.apache.nifi.processors.livy.ExecuteSparkInteractive.onTrigger(ExecuteSparkInteractive.java:197) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Created 06-11-2018 11:46 AM
Problem was in "sqlContext.registerDataFrameAsTable"
What investigate similar (HTTP) errors ?
Created 06-11-2018 01:31 PM
@Dmitro Vasilenko I would recommend you review Livy logs under /var/log/livy2 to understand why the application submission is failing.
HTH
Created 06-11-2018 01:45 PM
Hi! one more question, how control count "livy-session-"" ?
Created 07-31-2018 09:34 AM
Hi ,
Any solution for this. I am also facing same issue.
Created 07-31-2018 10:39 AM
Hi! Try One of this changes:
Zeppelin | NIFI ExecuteSparkInteractive | |
preds = spark.sql(‘select * from sandbox.CHURN_PRP_D_PREDS_S ’) | preds = spark.sql("select * from andbox.CHURN_PRP_D_PREDS_S") | |
dir = ‘ /user/CHURN_PRP_D’ | dir = "hdfs:/user/CHURN_PRP_D" | |
date_df = spark.sql("select to_date('{}') as score_date".format(date_calc)) | ttt =
spark.createDataFrame([(date_calc,)], ["t"])
date_df = ttt.select(to_date(ttt.t)) |
|
sqlContext.registerDataFrameAsTable(train, "train") | train.registerTempTable("train") |
Created 07-31-2018 11:49 AM
Hi,
I am trying to execute the below statements as part of code values in processor :
import org.apache.spark.sql.hive.HiveContext
val hiveContext = new HiveContext(sc)
import hiveContext._
hiveContext.sql("insert into cpv.del values('from livy')")
After looking recent comments from @Dmitro Vasilenko @, I have changed code to
spark.sql("insert into dbname.tablename values('from livy')")
Still Getting bad request error. Please look into attachements for configuration parameters. tempsnip.png tempsnipn.png
Looking for quick response, Thanks
Created 09-04-2018 03:18 PM
did u get a chance to look this. @nifi @ExecuteSparkInteractive @spark @Timothy Spann
spark.sql("insert into dbname.tablename values('from livy')") is not working
spark.sql("insert into dbname.tablename select * from dbname.tablename") is working
For more information, please look into my previous posts on same page/tasl/requiement.
@nifi
@executesparkintactive