Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Zeppelin fails - address is already in use

avatar
Expert Contributor

I have latest HDP 2.5 installed and controlled by Ambari. The Zepellin Notebook service never starts. Logs revealed that exception is thrown, which indicates "This address is already in use".

Some search on Internet revealed that problem may be caused by the fact that another program is using Zepellin's port 9995. However netstat command does not show anything using this port. Also I tried to change port to 9996 and it did not help either. It looks like the exception's message is misleading and problem is somewhere else.

Any help on this would be appreciated.

1 ACCEPTED SOLUTION

avatar
Expert Contributor

It turned out that problem was quite different. Actually Zeppelin service was working fine, Ambari simply did not report it properly. "Address is already in use" message was shown simply because I tried to start Zeppelin service again when another instance of it was already running (but nobody knew about it).

So this particular problem (Address is already in use) is now solved. However new problem started.

So I started a new thread regarding a different problem

View solution in original post

6 REPLIES 6

avatar

@Dmitry Otblesk Can you please share the logs ?

avatar
Expert Contributor

@Santhosh B Gowda

I no longer have access to previous log as I deleted Zeppelin service and tried to reinstall it again. During the installation it thrown an error. Below is the content of stderr output

Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py", line 330, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py", line 174, in start self.create_zeppelin_dir(params) File "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py", line 70, in create_zeppelin_dir recursive_chmod=True File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in action_create_on_execute self.action_delayed("create") File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456, in action_delayed self.get_hdfs_resource_executor().action_delayed(action_name, self) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 247, in action_delayed self._assert_valid() File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 231, in _assert_valid self.target_status = self._get_file_status(target) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 292, in _get_file_status list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 179, in run_command _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output raise ExecutionFailed(err_msg, code, files_output[0], files_output[1]) resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://lenu.dom.hdp:50070/webhdfs/v1/user/zeppelin?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpeaTzEm 2>/tmp/tmpwrNLnT' returned 7. curl: (7) Failed connect to lenu.dom.hdp:50070; Connection refused 000

avatar
Super Collaborator

Hi @Dmitry Otblesk

What exactly did you check with netstat?

If you've only checked what ports are used by listening services (netstat -l) I suggest to check all ports. I've seen cases when Hadoop services tried to listen on ports used as the source port by other TCP connections:

netstat -anp|grep 9995

avatar
Expert Contributor

@Alexandru Anghel

I tried the command you've suggested . It returned only 1 row

unix 3 [ ] STREAM CONNECTED 49995 5239/tracker-store

So, there is nothing on port 9995

avatar
Super Collaborator

Hmm, okay, but I expect this to be transient as we're talking about TCP outgoing connections. The connection that could have potentially caused the issue it's not there any longer.

So it's more important when you start Zeppelin and get the "This address is already in use" also to check the netstat at around the same time.

The other error you get when you tried to install Zeppelin again is not from Zeppelin but from Ambari trying to create the zeppelin user home folder in HDFS. It looks like HDFS (WebHDFS in this case) is not working so check that please (lenu.dom.hdp on port 50070).

avatar
Expert Contributor

It turned out that problem was quite different. Actually Zeppelin service was working fine, Ambari simply did not report it properly. "Address is already in use" message was shown simply because I tried to start Zeppelin service again when another instance of it was already running (but nobody knew about it).

So this particular problem (Address is already in use) is now solved. However new problem started.

So I started a new thread regarding a different problem