Data Nodes not starting

[#1]

During the testing the datanodes went down now they are not coming up I am not concerned about the data at this point just need to restart the system. following is the log

[#2]

2016-03-08 15:33:35,182 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485

2016-03-08 15:33:35,183 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,183 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,200 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,200 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,216 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,216 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,217 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,301 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-08 15:33:35,301 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,301 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,318 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,318 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,334 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,334 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,334 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,335 - Group['spark'] {}
2016-03-08 15:33:35,336 - Group['hadoop'] {}
2016-03-08 15:33:35,337 - Group['users'] {}
2016-03-08 15:33:35,337 - User['hive'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,337 - User['zookeeper'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,338 - User['oozie'] {'gid': 'hadoop', 'groups': ['users']}
2016-03-08 15:33:35,338 - User['ams'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,338 - User['tez'] {'gid': 'hadoop', 'groups': ['users']}
2016-03-08 15:33:35,339 - User['mahout'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,339 - User['spark'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,340 - User['ambari-qa'] {'gid': 'hadoop', 'groups': ['users']}
2016-03-08 15:33:35,340 - User['hdfs'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,341 - User['sqoop'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,341 - User['yarn'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,341 - User['mapred'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,342 - User['hbase'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,342 - User['hcat'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,343 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-03-08 15:33:35,344 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-03-08 15:33:35,347 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-03-08 15:33:35,347 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-03-08 15:33:35,348 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-03-08 15:33:35,349 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-03-08 15:33:35,352 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-03-08 15:33:35,352 - Group['hdfs'] {'ignore_failures': False}
2016-03-08 15:33:35,352 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
2016-03-08 15:33:35,353 - Directory['/etc/hadoop'] {'mode': 0755}
2016-03-08 15:33:35,362 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-03-08 15:33:35,362 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-03-08 15:33:35,371 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-03-08 15:33:35,383 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,384 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,385 - Changing owner for /var/run/hadoop from 509 to root
2016-03-08 15:33:35,385 - Changing group for /var/run/hadoop from 501 to root
2016-03-08 15:33:35,385 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,388 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,389 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,389 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-03-08 15:33:35,395 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,396 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-03-08 15:33:35,396 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-03-08 15:33:35,399 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-03-08 15:33:35,402 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-03-08 15:33:35,495 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-08 15:33:35,495 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,495 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,511 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,511 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,526 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,526 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,526 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,527 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-08 15:33:35,527 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,527 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,542 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,542 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,557 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,557 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,557 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,561 - Directory['/etc/security/limits.d'] {'owner': 'root', 'group': 'root', 'recursive': True}
2016-03-08 15:33:35,565 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2016-03-08 15:33:35,566 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,572 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2016-03-08 15:33:35,572 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,578 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,583 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2016-03-08 15:33:35,583 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,587 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,588 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,593 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2016-03-08 15:33:35,593 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,597 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,602 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2016-03-08 15:33:35,602 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,607 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,612 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2016-03-08 15:33:35,612 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,642 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2016-03-08 15:33:35,648 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2016-03-08 15:33:35,648 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,661 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,662 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0751, 'recursive': True}
2016-03-08 15:33:35,674 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/boot/efi', '/dstage', '/home', '/opt', '/tmp', '/usr', '/var', '/data1', '/data2', '/data3', '/data4', '/data5', '/data6', '/data7', '/data8', '/data9', '/data10', '/proc/sys/fs/binfmt_misc', '/bdata1', '/bdata2', '/bdata3', '/data4/ramdisk'].
2016-03-08 15:33:35,675 - Mount point for directory /bdata1/hadoop/hdfs/data is /
2016-03-08 15:33:35,675 - Mount point for directory /bdata2/hadoop/hdfs/data is /
2016-03-08 15:33:35,675 - Mount point for directory /bdata3/hadoop/hdfs/data is /
2016-03-08 15:33:35,675 - File['/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-03-08 15:33:35,676 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2016-03-08 15:33:35,676 - Changing owner for /var/run/hadoop from 0 to hdfs
2016-03-08 15:33:35,676 - Changing group for /var/run/hadoop from 0 to hadoop
2016-03-08 15:33:35,677 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2016-03-08 15:33:35,677 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2016-03-08 15:33:35,677 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2016-03-08 15:33:35,692 - Deleting File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid']
2016-03-08 15:33:35,692 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}

[#3]

Try

ls -l /usr/hdp/current/hadoop-client/conf/

Do you see config files there, like hdfs-site.xml etc? If not something is wrong. If installed by Ambari, /usr/hdp/current/hadoop-client/conf should be symlink to /etc/hadoop/<hdp-version>/0. If all that is fine, go to one DN, and check its logs. HTH.

[#4]

I got the same error and config files are there, could you help us to resolve the issue.

ls -l /usr/hdp/current/hadoop-client/conf/ total 196 -rw-r--r-- 1 hdfs hadoop 2069 May 17 16:07 capacity-scheduler.xml -rw-r--r-- 1 hdfs root 1020 May 17 15:43 commons-logging.properties -rw-r--r-- 1 hdfs hadoop 1335 Feb 11 13:15 configuration.xsl -rw-r--r-- 1 root hadoop 1019 May 17 15:18 container-executor.cfg -rw-r--r-- 1 hdfs hadoop 2535 May 17 16:07 core-site.xml -rw-r--r-- 1 root root 415 May 17 16:07 dfs_data_dir_mount.hist -rw-r--r-- 1 hdfs hadoop 1 May 17 15:57 dfs.exclude -rw-r--r-- 1 root root 3979 Feb 11 13:15 hadoop-env.cmd -rw-r--r-- 1 hdfs hadoop 5370 May 17 15:08 hadoop-env.sh -rw-r--r-- 1 hdfs root 899 May 17 15:43 hadoop-metrics2.properties -rw-r--r-- 1 root root 2490 Feb 11 13:15 hadoop-metrics.properties -rw-r--r-- 1 hdfs hadoop 1342 May 17 16:07 hadoop-policy.xml -rw-r--r-- 1 hdfs hadoop 6575 May 17 16:07 hdfs-site.xml -rw-r--r-- 1 hdfs root 1602 May 17 15:43 health_check -rw-r--r-- 1 root root 3518 Feb 11 13:15 kms-acls.xml -rw-r--r-- 1 root root 1527 Feb 11 13:15 kms-env.sh -rw-r--r-- 1 root root 1631 Feb 11 13:15 kms-log4j.properties -rw-r--r-- 1 root root 5511 Feb 11 13:15 kms-site.xml -rw-r--r-- 1 hdfs hadoop 8699 May 17 15:43 log4j.properties -rw-r--r-- 1 root root 951 Feb 11 13:15 mapred-env.cmd -rw-r--r-- 1 hdfs root 661 May 17 15:18 mapred-env.sh -rw-r--r-- 1 root root 4113 Feb 11 13:15 mapred-queues.xml.template -rw-r--r-- 1 mapred hadoop 6571 May 17 16:07 mapred-site.xml -rw-r--r-- 1 root root 758 Feb 11 13:15 mapred-site.xml.template drwxr-xr-x 2 root hadoop 4096 May 17 15:18 secure -rw-r--r-- 1 hdfs root 26 May 17 15:18 slaves -rw-r--r-- 1 hdfs hadoop 918 May 17 16:07 ssl-client.xml -rw-r--r-- 1 mapred hadoop 2316 Feb 11 13:15 ssl-client.xml.example -rw-r--r-- 1 hdfs hadoop 1034 May 17 16:07 ssl-server.xml -rw-r--r-- 1 mapred hadoop 2268 Feb 11 13:15 ssl-server.xml.example -rw-r--r-- 1 hdfs root 945 May 17 15:18 taskcontroller.cfg -rwxr-xr-x 1 root root 4221 May 17 15:43 task-log4j.properties -rw-r--r-- 1 hdfs hadoop 86 May 17 15:43 topology_mappings.data -rwxr-xr-x 1 root root 2358 May 17 15:43 topology_script.py -rw-r--r-- 1 root root 2250 Feb 11 13:15 yarn-env.cmd -rwxr-xr-x 1 yarn hadoop 4908 May 17 16:07 yarn-env.sh -rw-r--r-- 1 yarn hadoop 0 May 17 15:18 yarn.exclude -rw-r--r-- 1 yarn hadoop 14304 May 17 16:07 yarn-site.xml

[#5 · Accepted solution]

Issue is resolved after adding the parameter in

/etc/security/limits.d/hdfs.conf

hdfs - nofile 32768 hdfs - nproc 65536

And then restart the datanode.