Support Questions
Find answers, ask questions, and share your expertise

Installation of Livy for Spark2 Server in HDP3.0

Contributor

Hi, Could you please help me to know about the installation of "Livy for Spark2 Server" in HDP3.0. I've tried to follow link - https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.0/installing-spark/content/installing_spark_usin..., but not getting enough clue.

 

More detail - Wanted to integrate HDP 3.0 with KNIME (4.0.1) Analytics Platform. while using spark getting below error

image

 

below is the log file

2019-09-11 20:59:48,543 : ERROR : KNIME-Worker-3 : : GetContextsRequest : Create Spark Context (Jobserver) : 4:204 : HTTP Status code: 302 | Response Body: <html> <head> <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://sandbox-hdp.hortonworks.com:8088/proxy/redirect/application_1567855906129_0004/contexts">here...> </div> </body></html>
2019-09-11 20:59:48,545 : ERROR : KNIME-Worker-3 : : Node : Create Spark Context (Jobserver) : 4:204 : Execute failed: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
org.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:164)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:133)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:62)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:72)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:279)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.open(JobserverSparkContext.java:197)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:144)
at org.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:115)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)
at org.knime.core.node.Node.execute(Node.java:979)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2019-09-11 20:59:57,919 : ERROR : KNIME-Worker-4 : : GetContextsRequest : Create Spark Context (Jobserver) : 4:204 : HTTP Status code: 302 | Response Body: <html> <head> <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://sandbox-hdp.hortonworks.com:8088/proxy/redirect/application_1567855906129_0004/contexts">here...> </div> </body></html>
2019-09-11 20:59:57,920 : ERROR : KNIME-Worker-4 : : Node : Create Spark Context (Jobserver) : 4:204 : Execute failed: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
org.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:164)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:133)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:62)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:72)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:279)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.open(JobserverSparkContext.java:197)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:144)
at org.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:115)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)
at org.knime.core.node.Node.execute(Node.java:979)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2019-09-11 21:02:48,942 : ERROR : KNIME-Worker-5 : : GetContextsRequest : Create Spark Context (Jobserver) : 4:204 : HTTP Status code: 302 | Response Body: <html> <head> <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://sandbox-hdp.hortonworks.com:8088/proxy/redirect/application_1567855906129_0004/contexts">here...> </div> </body></html>
2019-09-11 21:02:48,942 : ERROR : KNIME-Worker-5 : : Node : Create Spark Context (Jobserver) : 4:204 : Execute failed: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
org.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:164)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:133)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:62)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:72)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:279)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.open(JobserverSparkContext.java:197)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:144)
at org.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:115)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)
at org.knime.core.node.Node.execute(Node.java:979)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

 

1 REPLY 1

Re: Installation of Livy for Spark2 Server in HDP3.0

Contributor
Hi Team, Did anyone get chance to look into it. Thanks and Regards.