Support Questions
Find answers, ask questions, and share your expertise

HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

When I go to Zeppelin at port 9995, I get a 503 error. From the Ambari dashboard, the zeppelin notebook appears to be running. Nothing wrto Zeppelin has gone smoothly for me over the past 24 hours. Ports 8080 and 8888 are fine and I have ssh'd into my vm fine. Initially after restart, I could get to zeppelin on 9995 but the maria_dev and admin passwords weren't accepted. I logged in as root and reset those passwords before looking again to see that Zeppelin was unreachable. Any ideas why?

8 REPLIES 8

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

Super Collaborator
@Colin Cunningham

Can you try renaming notebook directory, it should be under /usr/hdp/current/zeppelin-server. Restart the zeppelin server and see if it works now. It may be that one of the notebook got corrupted and that will cause zeppelin web UI to throw HTTP error 503.

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

@rguruvannagari can this be done from ambari UI. i don't want to repeat my predicament where ambari overwrote my changes? but i'll try this. thank you. - Colin

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

Super Collaborator

Restarting zeppelin server service should create a empty notebook directory. With previous notebook directory It could be that zeppelin was stopped while a notebook was still in running status, this would have corrupted the notebook and may cause 503 error with zeppelin. See if zeppelin server is accessible with empty notebook directory. If it is then you may copy the notebooks from old renamed zeppelin directory after finding the corrupted notebook (most probably last modified notebook could be corrupted)

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

@rguruvannagari for what it's worth below is the output from my "Check Zeppelin" from Ambari on the zeppelin service.

stderr: /var/lib/ambari-agent/data/errors-343.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-stderr: /var/lib/ambari-agent/data/errors-343.txt

Traceback (most recent call last):

File "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/service_check.py", line 40, in <module>

ZeppelinServiceCheck().execute()

File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute

method(env)

File "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/service_check.py", line 37, in service_check

logoutput=True)

File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__

self.env.run()

File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run

self.run_action(resource, action)

File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action

provider_action()

File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run

tries=self.resource.tries, try_sleep=self.resource.try_sleep)

File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner

result = function(command, **kwargs)

File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call

tries=tries, try_sleep=try_sleep)

File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper

result = _call(command, **kwargs_copy)

File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call

raise Fail(err_msg)

resource_management.core.exceptions.Fail: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

stdout: /var/lib/ambari-agent/data/output-343.txt

2017-02-04 04:48:02,074 - call['ambari-python-wrap /usr/bin/hdp-select status spark-client'] {'timeout': 20}

2017-02-04 04:48:02,097 - call returned (0, 'spark-client - 2.5.0.0-1245')

2017-02-04 04:48:02,099 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf

2017-02-04 04:48:02,099 - Execute['curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200'] {'logoutput': True, 'tries': 10, 'try_sleep': 3}

2017-02-04 04:48:02,428 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:05,448 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:08,464 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:11,480 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:14,497 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:17,515 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:20,627 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:23,644 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

2017-02-04 04:48:26,661 - Retrying after 3 seconds. Reason: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://sandbox.hortonworks.com:9995 | grep 200' returned 1.

Command failed after 1 tries

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

Super Collaborator

Is zeppelin service restarted after renaming /usr/hdp/current/zeppelin-server/notebook

#mv /usr/hdp/current/zeppelin-server/notebook /usr/hdp/current/zeppelin-server/notebook_old

Restart zeppelin service from ambari, it should create a new empty notebook directory.

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

@Colin Cunningham

I was working with the similar problem before. The problem was that the ownership of /usr/hdp/<VERSION>/zeppelin/webapps/ was set to root:root. I logged in to zeppelin host as root and ran:

$ chown -R zeppelin:hadoop /usr/hdp/<VERSION>/zeppelin/webapps/

Make sure its permission is 755.

Restart Zeppelin service and see if it helps.

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

thank you Daniel. i will keep this in mind but I just started over. so far so good, but this may very well have been the issue. thanks - colin

Re: HDP2.5 Sandbox up and zeppelin service running but now getting 503 error on port 9995

@Colin Cunningham

You are welcome. I have now created HCC for the same: https://community.hortonworks.com/articles/81471/zeppelin-ui-returns-503-error.html

Please, vote it if you like it.