Created 08-24-2018 08:54 PM
Hi,
I am using HDP3.0 and cannot start the livy server:
18/08/24 20:24:39 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead. 18/08/24 20:24:39 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead. 18/08/24 20:24:39 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: * 18/08/24 20:24:39 INFO LineBufferedStream: stdout: File "/usr/bin/hdp-select", line 251 18/08/24 20:24:39 INFO LineBufferedStream: stdout: print "ERROR: Invalid package - " + name 18/08/24 20:24:39 INFO LineBufferedStream: stdout: ^ 18/08/24 20:24:39 INFO LineBufferedStream: stdout: SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)? 18/08/24 20:24:39 INFO LineBufferedStream: stdout: ls: cannot access /usr/hdp//hadoop/lib: No such file or directory 18/08/24 20:24:39 INFO LineBufferedStream: stdout: Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx 18/08/24 20:24:39 INFO LineBufferedStream: stdout: at org.apache.spark.launcher.Main.main(Main.java:118) Exception in thread "main" java.lang.IllegalArgumentException: Fail to parse Spark version from is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx at org.apache.livy.utils.LivySparkUtils$.formatSparkVersion(LivySparkUtils.scala:155) at org.apache.livy.utils.LivySparkUtils$.testSparkVersion(LivySparkUtils.scala:82) at org.apache.livy.server.LivyServer.start(LivyServer.scala:74) at org.apache.livy.server.LivyServer$.main(LivyServer.scala:339) at org.apache.livy.server.LivyServer.main(LivyServer.scala)
The error looks like livy tried to use python3 to parse the python2 syntax. Any idea why this happens? Thanks.
Created 08-27-2018 04:49 AM
@Lian Jiang Looks like default python version is 3.0, you might want to change to 2.7.x.
Please refer supported python release: https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-installation/content/mmsr_softwar...
Created 08-27-2018 04:49 AM
@Lian Jiang Looks like default python version is 3.0, you might want to change to 2.7.x.
Please refer supported python release: https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-installation/content/mmsr_softwar...
Created 08-27-2018 05:03 AM
Hi @Lian Jiang,
Please see if this article can help you : https://community.hortonworks.com/content/supportkb/186304/unable-to-start-pyspark-jobs-when-running...
Please login and accept answer if it helped :)
Created 08-28-2018 04:21 PM
Thanks guys. Rolling back to python 2.7 made livy server succeed to start.