Need help please. I have used Ambari and HDP 2.3 and all the services got started manually the first time but then it's not starting. Not able to start data node or name node or secondary node.

[#1]

Below is the exception I am getting:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 433, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 102, in start
    namenode(action="start", hdfs_binary=hdfs_binary, upgrade_type=upgrade_type, env=env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 112, in namenode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 267, in service
    Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode'' returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-pp-hdp-m.out

[#2]

@Prakash Punj You accepted your own answer and I don't think this issue is resolved yet.

This is networking issue and I am curious to know what else is running in your private cloud and how netowkring is setup.

Ifconfig -a --> output please.

[#3]

@Neeraj Sabharwal

Output of ifconfig -a is below. I am sure this is a networking issue. I am very new to this environment, so having all kind of issue.

eth0      Link encap:Ethernet  HWaddr FA:16:3E:AB:2D:C1
          inet addr:10.0.7.1  Bcast:10.0.7.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:feab:2dc1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1454943 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1107795 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1997123550 (1.8 GiB)  TX bytes:150653800 (143.6 MiB)
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:975398 errors:0 dropped:0 overruns:0 frame:0
          TX packets:975398 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:383200056 (365.4 MiB)  TX bytes:383200056 (365.4 MiB)

[#4]

inet addr:10.0.7.1 - This is internal VM ip address dynamically assigned by VM hosts I think. VM's IP address is 192.168.25.39 and hostname is hdp-m.samitsolutions.com

Thanks

[#5]

Ok. hostname -f --> output

then ping hostname coming from about output

[#6]

hostname -f output is @Neeraj Sabharwal

hdp-m.samitsolutions.com

PING hdp-m.samitsolutions.com (192.168.25.139) 56(84) bytes of data.
64 bytes from hdp-m.samitsolutions.com (192.168.25.139): icmp_seq=1 ttl=63 time=0.190 ms
64 bytes from hdp-m.samitsolutions.com (192.168.25.139): icmp_seq=2 ttl=63 time=0.328 ms
64 bytes from hdp-m.samitsolutions.com (192.168.25.139): icmp_seq=3 ttl=63 time=0.409 ms
64 bytes from hdp-m.samitsolutions.com (192.168.25.139): icmp_seq=4 ttl=63 time=0.291 ms
64 bytes from hdp-m.samitsolutions.com (192.168.25.139): icmp_seq=5 ttl=63 time=0.198 ms

[#7]

Try configuring your environment using internal ip address(10.0.7.1) instead of 192.168.25.x one

[#8]

@Prakash Punj Here is the problem

Your hostname resolves to 192 address

add following entry in /etc/hosts and then give it a shot

192.168.25.139 hdp-m.samitsolutions.com

[#9]

@Neeraj SabharwalActually i do have the entry you mentioned:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

192.168.25.139 hdp-m.samitsolutions.com hdp-m

The other VM internal address you mentioned 10.0.71, per the VM support person, that's an internal address that VM dynamically get assigned and 192.168.x.x. is the address should be used.

Thanks

[#10]

@Prakash Punj

5:12:21,968 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.net.BindException: Port in use: pp-hdp-m:50070 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856) at

What is this host, pp-hdp-m?

[#11]

Thanks @Neeraj Sabharwal

I used internal IP and that did work. So thank you so much. Now I moved on other components. most of them got installed successfully except Ambari-Collector an Ambari-Monitor. I have started a different discussion thread on that. Again thank you so much. Next thing is I have to try HUE, any good documentation on that ? I want to use Ambari for that ..

Thanks