question Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node. in Support Questions

Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Fri, 05 Feb 2016 07:49:26 GMT

Below is the exception I am getting:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 433, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 102, in start
    namenode(action="start", hdfs_binary=hdfs_binary, upgrade_type=upgrade_type, env=env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 112, in namenode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 267, in service
    Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode'' returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-pp-hdp-m.out

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

afernandez — Fri, 05 Feb 2016 07:50:52 GMT

What's the output of /var/log/hadoop/hdfs/hadoop-hdfs-namenode-pp-hdp-m.log ?

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

SK1 — Fri, 05 Feb 2016 14:43:08 GMT

Can you please post logs from /var/log/hadoop/hdfs/hadoop-hdfs-namenode-pp-hdp-m.out. Also post ulimit -a from the nodes where DN and NN are running ?

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Sat, 06 Feb 2016 04:40:51 GMT

Hi ! Saurabh - Thank you so much. @Saurabh Kumar

output of /var/log/hadoop/hdfs/hadoop-hdfs-namenode-pp-hdp-m.out

ulimit -a for user hdfs

core file size (blocks, -c) unlimited

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 63413

max locked memory (kbytes, -l) 64

max memory size (kbytes, -m) unlimited

open files (-n) 128000

pipe size (512 bytes, -p) 8

POSIX message

queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 8192

cpu time (seconds, -t)

unlimited max user processes (-u) 65536

virtual memory (kbytes, -v)

unlimited file locks (-x) unlimited

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Sat, 06 Feb 2016 05:19:46 GMT

@Saurabh Kumar

Looks like what is happening is it's not able to assign port 50070 to the hostname.

java.net.BindException: Port in use: pp-hdp-m:50070

But when I checked this port is not used by any other process..

thanks

Prakash Punj

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

nsabharwal — Sat, 06 Feb 2016 05:32:20 GMT

@Prakash Punj

Are you using vm? Please provide more details on your environment. vagrant?

5:12:21,968 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.net.BindException: Port in use: pp-hdp-m:50070 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856) at

netstat -anp| grep 50070

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

Shelton — Sat, 06 Feb 2016 06:01:57 GMT

Kill the process that is using the port 50070

15:12:21,968 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.net.BindException: Port in use: pp-hdp-m:50070 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856) at

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Sat, 06 Feb 2016 06:02:42 GMT

Thanks @Neeraj Sabharwal. I am very new to to this. I am using HDP 2.3 on Ambari. Thanks for helping me out..

Yes I am using VM - centos7. Looks like something messed up on hostname configuration, Internal IP or something like that.

content of /etc/hosts file:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

192.168.24.116 pp-hdp-s2

192.168.24.117 pp-hdp-s1

192.168.24.118 pp-hdp-m - This is where I am installing Ambari and Namenode

Content of cat /etc/resolv.conf

; generated by /usr/sbin/dhclient-script

search ASOTC ( where is this asotc coming from as I do see this name getting appended in some of the hdfs.site.xml as host name)

nameserver 192.168.24.1

Below is one entry from netstat ( looks like VM has one more internal IP 10.0.2.14)

tcp 0 0 10.0.2.14:49100 pp-hdp-m: eforward TIME_WAIT

hostname -f

pp-hdp-m

hostname -i

192.168.24.118

NETWORK configuration file:

NETWORKING=yes

HOSTNAME=pp-hdp-m

NOZEROCONF=yes

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

Shelton — Sat, 06 Feb 2016 06:09:56 GMT

@Prakash Punj try rebooting that server ans restart the Ambari server.. Then log you upload says "Port in use: pp-hdp-m:50070 at org.apache.hadoop.http.HttpServer2.openListeners"

If you want help then you need to follow some tips from this forum by elimination we cn come to your rescue !

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

nsabharwal — Mon, 19 Aug 2019 09:57:45 GMT

@Prakash Punj

See this from my environment.

There is issue with the networking in your environment. You have to find out whats running in your environment and kill those processes.

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Sat, 06 Feb 2016 11:31:16 GMT

@Saurabh Kumar @Neeraj Sabharwal

I tried rebooting the server and infact tried to redo everything. Fixed the host-name issue but still not able to start the namenode. It's finding port 50070 already in use. I even tried changing this port to something but then it says that port is also in use. Below is what it says in the log file:

2016-02-06 00:47:14,934 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.net.BindException: Port in use: pp-hdp-m.asotc:50070

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

nsabharwal — Sat, 06 Feb 2016 11:38:27 GMT

@Prakash Punj

Whats the output of netstat -ant|grep 50070?

@Prakash Punj See this https://community.hortonworks.com/articles/14912/osquery-tool-to-troubleshoot-os-processes.html

Use the above to get more idea on processes running.

Look into the network configs of your hosts.

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

nsabharwal — Sat, 06 Feb 2016 11:39:30 GMT

@Prakash Punj This thread is very long. Please paste the output of netstat -anp | grep 50070

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

nsabharwal — Sat, 06 Feb 2016 12:05:57 GMT

@Prakash Punj See this https://community.hortonworks.com/articles/14912/osquery-tool-to-troubleshoot-os-processes.html

Use the above to get more idea on processes running.

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

Shelton — Mon, 19 Aug 2019 09:57:37 GMT

@Saurabh Kuma

I see you are installing Amabari from 192.168.24.118 pp-hdp-m but the output of your netstat shows this IP 172.24.64.98 using the port 50070.

Can you identify this host 172.24.64.98 ?

Typically the netstat command should have your hostname 4th and 6th line see the attached out put from my Ambari server. I guess youe network configuration is wrong.See my attached screenshot

My Ambarai server is listeneing to the correct port

Your hostname looks okay

1. Do the following

[root@had etc] # vi /etc/sysconfig/network

NETWORKING=yes

HOSTNAME= pp-hdp-m

2. Edit the ifcfg-eth0 - ifcfg-eth0 depending on attached NIC’s

[root@had ] vi /etc/sysconfig/network-scripts

DEVICE=eth0

HWADDR=00:15:17:12:F2:50

TYPE=Ethernet

UUID=6428569e-e729-4985-974d-e87fc81a8796

NM_CONTROLLED=yes

ONBOOT=yes

BOOTPROTO=dhcp

3. See the output of ifconfig check the IP addresses mapped to all the NIC's attached:

[root@had ] ifconfig

Your cuurent IP should map to one of the NIC’s attached

If you are on a vmware check the IP's of other hosts

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Mon, 08 Feb 2016 00:58:07 GMT

@Neeraj Sabharwal @Saurabh Kumar @

Issue is still there. Below is what i get when I try to start name node from Ambari and the namenode log still says "port in use 50070". Looks like there is something very basic that I am missing. I am very new to this so don't much. I tried reimage all the instance and gave all the devices new IP but still the same result.

I dont get anything when I try the command: netstat -anp | grep 50070

Thanks

resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode'' returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-hdp-m.out

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

nsabharwal — Mon, 08 Feb 2016 01:05:00 GMT

@Prakash Punj Are you running this in your laptop?

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Mon, 08 Feb 2016 01:25:51 GMT

I am using Ambari 2.2 with HDP 2.3 and getting the error when installing the hadoop services from Ambari. DataNode gets sttarted just fine but it's NameNode which is failing. It says port is in use 50070 there is nothing running on that port. I also tried changing the port to 50071 but the error remains the same. It then says port is in use 50071. So it's not a question of finding that process and killing. Something else is missing from the network configuration.

I am using VM.

Thanks

Prakash, Cary, USA

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

nsabharwal — Mon, 08 Feb 2016 01:32:17 GMT

@Prakash Punj I know that but where is VM? In your laptop?

Re: Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

prakashpunj — Mon, 08 Feb 2016 01:49:53 GMT

@Neeraj Sabharwal

I am running VM in a private cloud environment running centos 6.7 image. It's not on my laptop.

Thanks

Prakash