Member since
05-07-2016
3
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6142 | 05-13-2016 09:56 PM |
08-22-2016
09:53 AM
Hi @Estannislao Like I noted, I had to go back to spark 1.5.0. Next, I added the complete path to spark/bin to my PATH environment variable. I did not have to change anything in code. Regards, Bhushan
... View more
05-13-2016
09:56 PM
I made sparkR available in the path and it works. Also, had to go back to spark 1.5.0
... View more
05-13-2016
09:47 AM
Update: Going back to Spark 1.5.0 solved the second problem (I read on another post that 1.6.1 is not supported in notebook). But I still had to change the path to the complete path for sparkR.
... View more
05-09-2016
10:27 AM
Also, this happens in both yarn and process modes. Appreciate any help!
... View more
05-07-2016
04:28 PM
I am running hadoop 2.7.2 with spark 1.6.1 on Yarn. I have built hue and livy from the source at http://github.com/cloudera/hue. I built livy against spark 1.6.1 using: mvn -DskipTests -Dspark.version=1.6.1 clean package I am trying to run the Hue Notebook Sample (or my simple own code). I am able to run hive queries, pySpark, scala, but not R. I get the following error in the container log on R snippets: Exception thrown when initializing server: java.lang.Exception: Cannot find sparkR executable SPARK_HOME and other environment variables are set. I am also able to run sparkR shell on yarn or local from the terminal. I figured the exception was coming from hue/apps/spark/java/livy-repl/src/main/scala/com/cloudera/hue/livy/repl/sparkr/SparkRInterpreter.scala as it was not finding the executable. I replaced the "sparkR" with the complete path to sparkR to see what happens, built again, but then I get the following error in the container log, after waiting 10000ms for Spark Context to Initialize. Appreciate any hints on what might be going wrong? I do not see any other useful logs on yarn or spark. SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-root/nm-local-dir/usercache/root/filecache/18/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/mydh/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/05/07 15:59:55 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
16/05/07 15:59:56 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1462660966341_0003_000001
16/05/07 15:59:57 INFO spark.SecurityManager: Changing view acls to: root
16/05/07 15:59:57 INFO spark.SecurityManager: Changing modify acls to: root
16/05/07 15:59:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/05/07 15:59:57 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
16/05/07 15:59:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization
16/05/07 15:59:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 15:59:57 INFO util.log: Logging initialized @2184ms
16/05/07 15:59:57 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/05/07 15:59:57 INFO servlet.ScalatraListener: The cycle class name from the config: com.cloudera.hue.livy.repl.ScalatraBootstrap
16/05/07 15:59:57 INFO servlet.ScalatraListener: Initializing life cycle class: ScalatraBootstrap
16/05/07 15:59:57 ERROR sparkr.SparkRInterpreter: Process has died with 127
16/05/07 15:59:57 INFO repl.ScalatraBootstrap: Notifying http://127.0.0.1:8998/sessions/1/callback that we're up
16/05/07 15:59:57 INFO handler.ContextHandler: Started c.c.h.j.s.ServletContextHandler@168d645d{/,file:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1462660966341_0003/container_1462660966341_0003_01_000001/src/main/com/cloudera/hue/livy/repl,AVAILABLE}
16/05/07 15:59:57 INFO server.ServerConnector: Started ServerConnector@1bd85653{HTTP/1.1}{0.0.0.0:35014}
16/05/07 15:59:57 INFO server.Server: Started @2697ms
16/05/07 15:59:57 INFO livy.WebServer: Starting server on 35014
16/05/07 16:00:07 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:00:17 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:00:27 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:00:37 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:00:47 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:00:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:01:07 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:01:17 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:01:27 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
16/05/07 16:01:37 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.
16/05/07 16:01:37 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Timed out waiting for SparkContext.)
16/05/07 16:01:37 INFO util.ShutdownHookManager: Shutdown hook called
16/05/07 16:01:37 INFO server.ServerConnector: Stopped ServerConnector@1bd85653{HTTP/1.1}{0.0.0.0:0}
16/05/07 16:01:37 INFO servlet.ScalatraListener: Destroying life cycle class: com.cloudera.hue.livy.repl.ScalatraBootstrap
16/05/07 16:01:37 INFO handler.ContextHandler: Stopped c.c.h.j.s.ServletContextHandler@168d645d{/,file:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1462660966341_0003/container_1462660966341_0003_01_000001/src/main/com/cloudera/hue/livy/repl,UNAVAILABLE}
... View more
Labels: