Created 10-18-2016 10:07 PM
Log :
2016-10-18 13:58:22,747 - Group['hadoop'] {} 2016-10-18 13:58:22,749 - Group['users'] {} 2016-10-18 13:58:22,749 - User['hive'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:22,897 - User['zookeeper'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,046 - User['ams'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,168 - User['ambari-qa'] {'gid': 'hadoop', 'groups': ['users']} 2016-10-18 13:58:23,293 - User['tez'] {'gid': 'hadoop', 'groups': ['users']} 2016-10-18 13:58:23,464 - User['hdfs'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,756 - User['sqoop'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:23,905 - User['yarn'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,071 - User['hcat'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,223 - User['mapred'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,388 - User['hbase'] {'gid': 'hadoop', 'groups': ['hadoop']} 2016-10-18 13:58:24,547 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-10-18 13:58:24,548 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2016-10-18 13:58:24,552 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2016-10-18 13:58:24,553 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'} 2016-10-18 13:58:24,554 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-10-18 13:58:24,555 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2016-10-18 13:58:24,559 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if 2016-10-18 13:58:24,559 - Group['hdfs'] {'ignore_failures': False} 2016-10-18 13:58:24,559 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']} 2016-10-18 13:58:24,698 - Directory['/etc/hadoop'] {'mode': 0755} 2016-10-18 13:58:24,709 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2016-10-18 13:58:24,710 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777} 2016-10-18 13:58:24,722 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2016-10-18 13:58:24,728 - Skipping Execute[('setenforce', '0')] due to not_if 2016-10-18 13:58:24,728 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,731 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,731 - Changing owner for /var/run/hadoop from 54372 to root 2016-10-18 13:58:24,731 - Changing group for /var/run/hadoop from 54343 to root 2016-10-18 13:58:24,731 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,735 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:24,737 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:24,737 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2016-10-18 13:58:24,744 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:24,745 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2016-10-18 13:58:24,746 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2016-10-18 13:58:24,750 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'} 2016-10-18 13:58:24,754 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2016-10-18 13:58:24,904 - Directory['/etc/security/limits.d'] {'owner': 'root', 'group': 'root', 'recursive': True} 2016-10-18 13:58:24,910 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2016-10-18 13:58:24,911 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,919 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml 2016-10-18 13:58:24,919 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,927 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,934 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml 2016-10-18 13:58:24,934 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,939 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-10-18 13:58:24,940 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,947 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml 2016-10-18 13:58:24,947 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,952 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,960 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml 2016-10-18 13:58:24,960 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:24,966 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-10-18 13:58:24,972 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml 2016-10-18 13:58:24,973 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-10-18 13:58:25,009 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...} 2016-10-18 13:58:25,018 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml 2016-10-18 13:58:25,018 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} 2016-10-18 13:58:25,034 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'} 2016-10-18 13:58:25,035 - Directory['/opt/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,035 - Directory['/tmp/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,036 - Directory['/var/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,036 - Directory['/var/log/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2016-10-18 13:58:25,036 - Directory['/var/log/audit/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'recursive': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'} 2016-10-18 13:58:25,037 - Ranger admin not installed /opt/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /tmp/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /var/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /var/log/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted /var/log/audit/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted 2016-10-18 13:58:25,037 - Directory['/opt/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/tmp/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/var/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/var/log/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,038 - Directory['/var/log/audit/hadoop/hdfs/namenode/namenode-formatted/'] {'recursive': True} 2016-10-18 13:58:25,040 - File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'), 'group': 'hadoop'} 2016-10-18 13:58:25,040 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755} 2016-10-18 13:58:25,041 - Changing owner for /var/run/hadoop from 0 to hdfs 2016-10-18 13:58:25,041 - Changing group for /var/run/hadoop from 0 to hadoop 2016-10-18 13:58:25,041 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True} 2016-10-18 13:58:25,041 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True} 2016-10-18 13:58:25,042 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'} 2016-10-18 13:58:25,046 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
Created 10-19-2016 07:02 PM
I have installed and uninstalled ambari so many times coz of that UUID and clusterd id is different from namenode and datanodes. thats why namenode is not started and datanodes too.
So I did
rm -rf /opt/hadoop/hdfs/data/*
/tmp/hadoop/hdfs/data/*,
/var/hadoop/hdfs/data,
/var/log/hadoop/hdfs/data,
/var/log/audit/hadoop/hdfs/data/*
sudo -u hdfs
/usr/hdp/2.3.6.0-3796/hadoop/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start namenode
su root
/usr/lib/hadoop/bin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode
By doing this I fixed my issue. Hopefull it will help someone.
Thanks
Created 10-18-2016 11:07 PM
Created 10-18-2016 11:46 PM
@mqureshi /var/log/ambari-server
Created 10-18-2016 11:50 PM
Those are Ambari server logs. If your namenode is down, then can you please share namenode logs? Should be under /var/log/hadoop
Created 10-19-2016 07:02 PM
I have installed and uninstalled ambari so many times coz of that UUID and clusterd id is different from namenode and datanodes. thats why namenode is not started and datanodes too.
So I did
rm -rf /opt/hadoop/hdfs/data/*
/tmp/hadoop/hdfs/data/*,
/var/hadoop/hdfs/data,
/var/log/hadoop/hdfs/data,
/var/log/audit/hadoop/hdfs/data/*
sudo -u hdfs
/usr/hdp/2.3.6.0-3796/hadoop/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start namenode
su root
/usr/lib/hadoop/bin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode
By doing this I fixed my issue. Hopefull it will help someone.
Thanks