question Re: Problem setting up Hue Spark notebook via livy server in Archives of Support Questions (Read Only)

Problem setting up Hue Spark notebook via livy server

vganji — Tue, 21 Apr 2026 13:54:15 GMT

Hello guys, I am trying to setup livy_server for the spark notebooks to work with Hue 3.9.0 with CDH 5.6.0. I am facing the following error:

livy server is failing to start with hue database password error:

[root@xxxxxx hue]# ./build/env/bin/hue livy_server
Traceback (most recent call last):
  File "./build/env/bin/hue", line 12, in <module>
    load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry
    execute_from_command_line(sys.argv)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 399, in e                                                            xecute_from_command_line
    utility.execute()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 392, in e                                                            xecute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 261, in f                                                            etch_command
    commands = get_commands()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 107, in g                                                            et_commands
    apps = settings.INSTALLED_APPS
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 49, in _setup
    self._wrapped = Settings(settings_module)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 128, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/utils/importlib.py", line 40, in import_modu                                                            le
    __import__(name)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/settings.py", line 309, in <module>
    "PASSWORD" : desktop.conf.get_database_password(),
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/conf.py", line 1219, in get_database_password
    password = DATABASE.PASSWORD_SCRIPT.get()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 140, in get
    return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 256, in get_value
    return self._coerce_type(raw_val, prefix)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 276, in _coerce_type
    return self.type(raw)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/conf.py", line 66, in coerce_password_from_script
    raise subprocess.CalledProcessError(p.returncode, script)
subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/4295-hue-HUE_SERVER/altscript.sh sec-8-password' returned non-zero exit status 1

I made sure current hue password set in CM is working fine.

Please help here.

Re: Problem setting up Hue Spark notebook via livy server

Mouna — Fri, 04 Mar 2016 19:51:50 GMT

Hello, Even im facing the same issue.Could anyone please help .

Re: Problem setting up Hue Spark notebook via livy server

vganji — Sun, 06 Mar 2016 02:34:51 GMT

This is now fixed.

Re: Problem setting up Hue Spark notebook via livy server

cjervis — Sun, 06 Mar 2016 13:42:51 GMT

Congratulations on solving your issue @vganji. Would you mind sharing your solution in case it can assist others?

Re: Problem setting up Hue Spark notebook via livy server

digitday — Thu, 31 Mar 2016 12:48:05 GMT

Hi all,
I've enabled spark notebook on Hue 3.9 in my CDH 5.5.2 cluster.
I've started LivyServer in this way:

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export PATH=$PATH:$SPARK_HOME/bin
/opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue livy_server&

Then, in a new PySpark notebook I've put this code:

a = sc.textFile("/user/admin/index_data.csv")
a.count()

and I've got this error:

05:43:58.141 [qtp455119019-16] ERROR c.c.hue.livy.server.SessionServlet$ - internal error
java.lang.IllegalStateException: Session is in state error
at com.cloudera.hue.livy.server.interactive.InteractiveWebSession.ensureRunning(InteractiveWebSession.scala:214) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.interactive.InteractiveWebSession.executeStatement(InteractiveWebSession.scala:73) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.interactive.InteractiveSessionServlet$$anonfun$16.apply(InteractiveSessionServlet.scala:128) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[livy-assembly.jar:na]
at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.withRouteMultiParams(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) [livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.org$scalatra$json$JsonSupport$$super$invoke(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.json.JsonSupport$$anonfun$invoke$1.apply(JsonSupport.scala:74) ~[livy-assembly.jar:na]
at org.scalatra.json.JsonSupport$$anonfun$invoke$1.apply(JsonSupport.scala:68) ~[livy-assembly.jar:na]
at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.withRouteMultiParams(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.json.JsonSupport$class.invoke(JsonSupport.scala:68) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.invoke(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) ~[livy-assembly.jar:na]
at scala.Option.flatMap(Option.scala:170) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) ~[livy-assembly.jar:na]
at scala.collection.immutable.Stream.flatMap(Stream.scala:442) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [livy-assembly.jar:na]
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) [livy-assembly.jar:na]
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) [livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) [livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.handle(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) [livy-assembly.jar:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.servlet.ServletHolder.handle(ServletHolder.java:808) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.Server.handle(Server.java:497) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.HttpChannel.handle(HttpChannel.java:310) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [livy-assembly.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
127.0.0.1 - - - 31/Mar/2016:05:43:58 -0700 "POST /sessions/0/statements HTTP/1.1" 500 58

Could you help me?
Thanks a lot

Re: Problem setting up Hue Spark notebook via livy server

bramd — Fri, 25 Nov 2016 18:53:36 GMT

Vganji,

How did you solve it?

Thanks,

Bramd

Re: Problem setting up Hue Spark notebook via livy server

bgooley — Sat, 26 Nov 2016 02:54:11 GMT

In Cloudera Manager 5.5 and higher, service passwords are stored in Cloudera Manager so that they are not exposed in the process directories. As part of this, the passwords are stored in a jceks file in each process directory. A password is then passed to the start scripts to decrypt the file containing the passwords.

When attempting to run the "hue" command from the command line, you will get the following:

File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 276, in _coerce_type
return self.type(raw)
File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/conf.py", line 66, in coerce_password_from_script
raise subprocess.CalledProcessError(p.returncode, script)
subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/4295-hue-HUE_SERVER/altscript.sh sec-8-password' returned non-zero exit status 1

This is because "altscript.sh" was not passed the password from CM.

To help allow you to be able to run commands from the commandline, I suggest trying:

# export HUE_CONF_DIR=/var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep HUE_SERVER| tail -1`

# HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 HUE_DATABASE_PASSWORD=password /opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue livy_server

Since the Spark Notebook is not supported in CDH yet, we cannot guarantee it will work as desired, but at the least, the steps I give should help you get around the exception that @vganji mentioned.

Re: Problem setting up Hue Spark notebook via livy server

bramd — Wed, 30 Nov 2016 16:28:42 GMT

I was still stuck with not knowing the HUE_DATABASE_PASSWORD

You can get it in cleartext from the CM REST API.

Make sure you're logged in to CM with admin rights.

http://:7180/api/v3/clusters//services//config?view=full

HTH
Bramd