Unable to start the Namenode, HDP 2.2.3

kkum — Wed, 19 Oct 2016 05:07:04 GMT

Log :

2016-10-18 13:58:22,747 - Group['hadoop'] {} 2016-10-18 13:58:22,749 - Group['users'] {} 2016-10-18 13:58:22,749 - User['hive'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:22,897 - User['zookeeper'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,046 - User['ams'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,168 - User['ambari-qa'] {'gid': 'hadoop', 'groups': ['users']} 2016-10-18 13:58:23,293 - User['tez'] {'gid': 'hadoop', 'groups': ['users']} 2016-10-18 13:58:23,464 - User['hdfs'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,756 - User['sqoop'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,905 - User['yarn'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,071 - User['hcat'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,223 - User['mapred'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,388 - User['hbase'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,547 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-10-18 13:58:24,548 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2016-10-18 13:58:24,552 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2016-10-18 13:58:24,553 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'} 2016-10-18 13:58:24,554 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-10-18 13:58:24,555 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2016-10-18 13:58:24,559 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if 2016-10-18 13:58:24,559 - Group['hdfs'] {'ignore_failures': False} 2016-10-18 13:58:24,559 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']} 2016-10-18 13:58:24,698 - Directory['/etc/hadoop'] {'mode': 0755} 2016-10-18 13:58:24,709 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2016-10-18 13:58:24,710 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777} 2016-10-18 13:58:24,722 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2016-10-18 13:58:24,728 - Skipping Execute[('setenforce', '0')] due to not_if 2016-10-18 13:58:24,728 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,731 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,731 - Changing owner for /var/run/hadoop from 54372 to root 2016-10-18 13:58:24,731 - Changing group for /var/run/hadoop from 54343 to root 2016-10-18 13:58:24,731 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,735 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:24,737 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:24,737 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2016-10-18 13:58:24,744 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:24,745 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2016-10-18 13:58:24,746 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2016-10-18 13:58:24,750 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'} 2016-10-18 13:58:24,754 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2016-10-18 13:58:24,904 - Directory['/etc/security/limits.d'] {'owner': 'root', 'group': 'root', 'recursive': True} 2016-10-18 13:58:24,910 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2016-10-18 13:58:24,911 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,919 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml 2016-10-18 13:58:24,919 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,927 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,934 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml 2016-10-18 13:58:24,934 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,939 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,940 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,947 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml 2016-10-18 13:58:24,947 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,952 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,960 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml 2016-10-18 13:58:24,960 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,966 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,972 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml 2016-10-18 13:58:24,973 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:25,009 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...} 2016-10-18 13:58:25,018 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml 2016-10-18 13:58:25,018 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} 2016-10-18 13:58:25,034 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:25,035 - Directory['/opt/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,035 - Directory['/tmp/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,036 - Directory['/var/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,036 - Directory['/var/log/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,036 - Directory['/var/log/audit/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'recursive': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'} 2016-10-18 13:58:25,037 - Ranger admin not installed /opt/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /tmp/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /var/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /var/log/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /var/log/audit/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted 2016-10-18 13:58:25,037 - Directory['/opt/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/tmp/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/var/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/var/log/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/var/log/audit/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,040 - File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'), 'group': 'hadoop'} 2016-10-18 13:58:25,040 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755} 2016-10-18 13:58:25,041 - Changing owner for /var/run/hadoop from 0 to hdfs 2016-10-18 13:58:25,041 - Changing group for /var/run/hadoop from 0 to hadoop 2016-10-18 13:58:25,041 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True} 2016-10-18 13:58:25,041 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True} 2016-10-18 13:58:25,042 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'} 2016-10-18 13:58:25,046 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}

Re: Unable to start the Namenode, HDP 2.2.3

mqureshi — Wed, 19 Oct 2016 06:07:20 GMT

@kiran thella

Which file is this log from? Is this the file under /var/log/hadoop/ folder?

Thanks

Re: Unable to start the Namenode, HDP 2.2.3

kkum — Wed, 19 Oct 2016 06:46:21 GMT

@mqureshi /var/log/ambari-server

Re: Unable to start the Namenode, HDP 2.2.3

mqureshi — Wed, 19 Oct 2016 06:50:26 GMT

Those are Ambari server logs. If your namenode is down, then can you please share namenode logs? Should be under /var/log/hadoop

Re: Unable to start the Namenode, HDP 2.2.3

kkum — Thu, 20 Oct 2016 02:02:35 GMT

I have installed and uninstalled ambari so many times coz of that UUID and clusterd id is different from namenode and datanodes. thats why namenode is not started and datanodes too.

So I did

rm -rf /opt/hadoop/hdfs/data/*

/tmp/hadoop/hdfs/data/*,

/var/hadoop/hdfs/data,

/var/log/hadoop/hdfs/data,

/var/log/audit/hadoop/hdfs/data/*

sudo -u hdfs

/usr/hdp/2.3.6.0-3796/hadoop/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start namenode

su root

/usr/lib/hadoop/bin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode

By doing this I fixed my issue. Hopefull it will help someone.

Thanks

question Re: Unable to start the Namenode, HDP 2.2.3 in Archives of Support Questions (Read Only)

Unable to start the Namenode, HDP 2.2.3

Re: Unable to start the Namenode, HDP 2.2.3

Re: Unable to start the Namenode, HDP 2.2.3

Re: Unable to start the Namenode, HDP 2.2.3

Re: Unable to start the Namenode, HDP 2.2.3