Problem setting up Hue Spark notebook via livy server

[#1]

Hello guys, I am trying to setup livy_server for the spark notebooks to work with Hue 3.9.0 with CDH 5.6.0. I am facing the following error:

livy server is failing to start with hue database password error:

[root@xxxxxx hue]# ./build/env/bin/hue livy_server
Traceback (most recent call last):
  File "./build/env/bin/hue", line 12, in <module>
    load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry
    execute_from_command_line(sys.argv)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 399, in e                                                            xecute_from_command_line
    utility.execute()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 392, in e                                                            xecute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 261, in f                                                            etch_command
    commands = get_commands()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 107, in g                                                            et_commands
    apps = settings.INSTALLED_APPS
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 49, in _setup
    self._wrapped = Settings(settings_module)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 128, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/utils/importlib.py", line 40, in import_modu                                                            le
    __import__(name)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/settings.py", line 309, in <module>
    "PASSWORD" : desktop.conf.get_database_password(),
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/conf.py", line 1219, in get_database_password
    password = DATABASE.PASSWORD_SCRIPT.get()
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 140, in get
    return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 256, in get_value
    return self._coerce_type(raw_val, prefix)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 276, in _coerce_type
    return self.type(raw)
  File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/conf.py", line 66, in coerce_password_from_script
    raise subprocess.CalledProcessError(p.returncode, script)
subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/4295-hue-HUE_SERVER/altscript.sh sec-8-password' returned non-zero exit status 1

I made sure current hue password set in CM is working fine.

Please help here.

[#2]

Hello, Even im facing the same issue.Could anyone please help .

[#3 · Accepted solution]

This is now fixed.

[#4]

Congratulations on solving your issue @vganji. Would you mind sharing your solution in case it can assist others?

[#5]

Hi all,
I've enabled spark notebook on Hue 3.9 in my CDH 5.5.2 cluster.
I've started LivyServer in this way:

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export PATH=$PATH:$SPARK_HOME/bin
/opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue livy_server&


Then, in a new PySpark notebook I've put this code:

a = sc.textFile("/user/admin/index_data.csv")
a.count()

and I've got this error:

05:43:58.141 [qtp455119019-16] ERROR c.c.hue.livy.server.SessionServlet$ - internal error
java.lang.IllegalStateException: Session is in state error
at com.cloudera.hue.livy.server.interactive.InteractiveWebSession.ensureRunning(InteractiveWebSession.scala:214) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.interactive.InteractiveWebSession.executeStatement(InteractiveWebSession.scala:73) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.interactive.InteractiveSessionServlet$$anonfun$16.apply(InteractiveSessionServlet.scala:128) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[livy-assembly.jar:na]
at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.withRouteMultiParams(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) [livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.org$scalatra$json$JsonSupport$$super$invoke(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.json.JsonSupport$$anonfun$invoke$1.apply(JsonSupport.scala:74) ~[livy-assembly.jar:na]
at org.scalatra.json.JsonSupport$$anonfun$invoke$1.apply(JsonSupport.scala:68) ~[livy-assembly.jar:na]
at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.withRouteMultiParams(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.json.JsonSupport$class.invoke(JsonSupport.scala:68) ~[livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.invoke(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) ~[livy-assembly.jar:na]
at scala.Option.flatMap(Option.scala:170) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) ~[livy-assembly.jar:na]
at scala.collection.immutable.Stream.flatMap(Stream.scala:442) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [livy-assembly.jar:na]
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) [livy-assembly.jar:na]
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) [livy-assembly.jar:na]
at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) [livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) [livy-assembly.jar:na]
at com.cloudera.hue.livy.server.SessionServlet.handle(SessionServlet.scala:33) [livy-assembly.jar:na]
at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) [livy-assembly.jar:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.servlet.ServletHolder.handle(ServletHolder.java:808) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.Server.handle(Server.java:497) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.HttpChannel.handle(HttpChannel.java:310) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [livy-assembly.jar:na]
at com.cloudera.hue.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [livy-assembly.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
127.0.0.1 - - - 31/Mar/2016:05:43:58 -0700 "POST /sessions/0/statements HTTP/1.1" 500 58


Could you help me?
Thanks a lot

[#6]

Vganji,

How did you solve it?

Thanks,

Bramd

[#7]

In Cloudera Manager 5.5 and higher, service passwords are stored in Cloudera Manager so that they are not exposed in the process directories.  As part of this, the passwords are stored in a jceks file in each process directory.  A password is then passed to the start scripts to decrypt the file containing the passwords.

When attempting to run the "hue" command from the command line, you will get the following:

File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/lib/conf.py", line 276, in _coerce_type
return self.type(raw)
File "/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hue/desktop/core/src/desktop/conf.py", line 66, in coerce_password_from_script
raise subprocess.CalledProcessError(p.returncode, script)
subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/4295-hue-HUE_SERVER/altscript.sh sec-8-password' returned non-zero exit status 1

This is because "altscript.sh" was not passed the password from CM.

To help allow you to be able to run commands from the commandline, I suggest trying:

# export HUE_CONF_DIR=/var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep HUE_SERVER| tail -1`

# HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 HUE_DATABASE_PASSWORD=password /opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue livy_server

Since the Spark Notebook is not supported in CDH yet, we cannot guarantee it will work as desired, but at the least, the steps I give should help you get around the exception that @vganji mentioned.

[#8]

I was still stuck with not knowing the HUE_DATABASE_PASSWORD

You can get it in cleartext from the CM REST API.

Make sure you're logged in to CM with admin rights.

http://:7180/api/v3/clusters//services//config?view=full

HTH
Bramd