HDP3.0: livy server cannot start

[#1]

Hi,

I am using HDP3.0 and cannot start the livy server:

18/08/24 20:24:39 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
18/08/24 20:24:39 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
18/08/24 20:24:39 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: *
18/08/24 20:24:39 INFO LineBufferedStream: stdout:   File "/usr/bin/hdp-select", line 251
18/08/24 20:24:39 INFO LineBufferedStream: stdout:     print "ERROR: Invalid package - " + name
18/08/24 20:24:39 INFO LineBufferedStream: stdout:                                     ^
18/08/24 20:24:39 INFO LineBufferedStream: stdout: SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)?
18/08/24 20:24:39 INFO LineBufferedStream: stdout: ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
18/08/24 20:24:39 INFO LineBufferedStream: stdout: Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
18/08/24 20:24:39 INFO LineBufferedStream: stdout:      at org.apache.spark.launcher.Main.main(Main.java:118)
Exception in thread "main" java.lang.IllegalArgumentException: Fail to parse Spark version from is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
        at org.apache.livy.utils.LivySparkUtils$.formatSparkVersion(LivySparkUtils.scala:155)
        at org.apache.livy.utils.LivySparkUtils$.testSparkVersion(LivySparkUtils.scala:82)
        at org.apache.livy.server.LivyServer.start(LivyServer.scala:74)
        at org.apache.livy.server.LivyServer$.main(LivyServer.scala:339)
        at org.apache.livy.server.LivyServer.main(LivyServer.scala)

The error looks like livy tried to use python3 to parse the python2 syntax. Any idea why this happens? Thanks.

[#2 · Accepted solution]

@Lian Jiang Looks like default python version is 3.0, you might want to change to 2.7.x.

Please refer supported python release: https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-installation/content/mmsr_softwar...

[#3]

Hi @Lian Jiang,

Please see if this article can help you : https://community.hortonworks.com/content/supportkb/186304/unable-to-start-pyspark-jobs-when-running...

Please login and accept answer if it helped :)

[#4]

Thanks guys. Rolling back to python 2.7 made livy server succeed to start.