Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Installation of Livy for Spark2 Server in HDP3.0

Highlighted

Installation of Livy for Spark2 Server in HDP3.0

Hi, Could you please help me to know about the installation of "Livy for Spark2 Server" in HDP3.0. I've tried to follow link - https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.0/installing-spark/content/installing_spark_usin..., but not getting enough clue.

 

More detail - Wanted to integrate HDP 3.0 with KNIME (4.0.1) Analytics Platform. while using spark getting below error

image

 

below is the log file

2019-09-11 20:59:48,543 : ERROR : KNIME-Worker-3 : : GetContextsRequest : Create Spark Context (Jobserver) : 4:204 : HTTP Status code: 302 | Response Body: <html> <head> <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://sandbox-hdp.hortonworks.com:8088/proxy/redirect/application_1567855906129_0004/contexts">here...> </div> </body></html>
2019-09-11 20:59:48,545 : ERROR : KNIME-Worker-3 : : Node : Create Spark Context (Jobserver) : 4:204 : Execute failed: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
org.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:164)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:133)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:62)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:72)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:279)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.open(JobserverSparkContext.java:197)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:144)
at org.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:115)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)
at org.knime.core.node.Node.execute(Node.java:979)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2019-09-11 20:59:57,919 : ERROR : KNIME-Worker-4 : : GetContextsRequest : Create Spark Context (Jobserver) : 4:204 : HTTP Status code: 302 | Response Body: <html> <head> <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://sandbox-hdp.hortonworks.com:8088/proxy/redirect/application_1567855906129_0004/contexts">here...> </div> </body></html>
2019-09-11 20:59:57,920 : ERROR : KNIME-Worker-4 : : Node : Create Spark Context (Jobserver) : 4:204 : Execute failed: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
org.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:164)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:133)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:62)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:72)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:279)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.open(JobserverSparkContext.java:197)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:144)
at org.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:115)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)
at org.knime.core.node.Node.execute(Node.java:979)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2019-09-11 21:02:48,942 : ERROR : KNIME-Worker-5 : : GetContextsRequest : Create Spark Context (Jobserver) : 4:204 : HTTP Status code: 302 | Response Body: <html> <head> <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://sandbox-hdp.hortonworks.com:8088/proxy/redirect/application_1567855906129_0004/contexts">here...> </div> </body></html>
2019-09-11 21:02:48,942 : ERROR : KNIME-Worker-5 : : Node : Create Spark Context (Jobserver) : 4:204 : Execute failed: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
org.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response For details see View > Open KNIME log.. Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:164)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:133)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:62)
at org.knime.bigdata.spark.core.sparkjobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
at org.knime.bigdata.spark.core.sparkjobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:72)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:279)
at org.knime.bigdata.spark.core.sparkjobserver.context.JobserverSparkContext.open(JobserverSparkContext.java:197)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:144)
at org.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:115)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)
at org.knime.core.node.Node.execute(Node.java:979)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

 

1 REPLY 1

Re: Installation of Livy for Spark2 Server in HDP3.0

Hi Team, Did anyone get chance to look into it. Thanks and Regards.