Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Data Nodes not starting

avatar
Contributor

During the testing the datanodes went down now they are not coming up I am not concerned about the data at this point just need to restart the system. following is the log

1 ACCEPTED SOLUTION

avatar

Issue is resolved after adding the parameter in

/etc/security/limits.d/hdfs.conf

hdfs - nofile 32768 hdfs - nproc 65536

And then restart the datanode.

View solution in original post

4 REPLIES 4

avatar
Contributor

2016-03-08 15:33:35,182 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485

2016-03-08 15:33:35,183 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,183 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,200 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,200 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,216 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,216 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,217 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,301 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-08 15:33:35,301 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,301 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,318 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,318 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,334 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,334 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,334 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,335 - Group['spark'] {}
2016-03-08 15:33:35,336 - Group['hadoop'] {}
2016-03-08 15:33:35,337 - Group['users'] {}
2016-03-08 15:33:35,337 - User['hive'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,337 - User['zookeeper'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,338 - User['oozie'] {'gid': 'hadoop', 'groups': ['users']}
2016-03-08 15:33:35,338 - User['ams'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,338 - User['tez'] {'gid': 'hadoop', 'groups': ['users']}
2016-03-08 15:33:35,339 - User['mahout'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,339 - User['spark'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,340 - User['ambari-qa'] {'gid': 'hadoop', 'groups': ['users']}
2016-03-08 15:33:35,340 - User['hdfs'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,341 - User['sqoop'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,341 - User['yarn'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,341 - User['mapred'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,342 - User['hbase'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,342 - User['hcat'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-03-08 15:33:35,343 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-03-08 15:33:35,344 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-03-08 15:33:35,347 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-03-08 15:33:35,347 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-03-08 15:33:35,348 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-03-08 15:33:35,349 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-03-08 15:33:35,352 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-03-08 15:33:35,352 - Group['hdfs'] {'ignore_failures': False}
2016-03-08 15:33:35,352 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
2016-03-08 15:33:35,353 - Directory['/etc/hadoop'] {'mode': 0755}
2016-03-08 15:33:35,362 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-03-08 15:33:35,362 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-03-08 15:33:35,371 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-03-08 15:33:35,383 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,384 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,385 - Changing owner for /var/run/hadoop from 509 to root
2016-03-08 15:33:35,385 - Changing group for /var/run/hadoop from 501 to root
2016-03-08 15:33:35,385 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,388 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,389 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,389 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-03-08 15:33:35,395 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,396 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-03-08 15:33:35,396 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-03-08 15:33:35,399 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-03-08 15:33:35,402 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-03-08 15:33:35,495 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-08 15:33:35,495 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,495 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,511 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,511 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,526 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,526 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,526 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,527 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-08 15:33:35,527 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-08 15:33:35,527 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-08 15:33:35,542 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-08 15:33:35,542 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-08 15:33:35,557 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-08 15:33:35,557 - Ensuring that hadoop has the correct symlink structure
2016-03-08 15:33:35,557 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-08 15:33:35,561 - Directory['/etc/security/limits.d'] {'owner': 'root', 'group': 'root', 'recursive': True}
2016-03-08 15:33:35,565 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2016-03-08 15:33:35,566 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,572 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2016-03-08 15:33:35,572 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,578 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,583 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2016-03-08 15:33:35,583 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,587 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-03-08 15:33:35,588 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,593 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2016-03-08 15:33:35,593 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,597 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,602 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2016-03-08 15:33:35,602 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,607 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-03-08 15:33:35,612 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2016-03-08 15:33:35,612 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,642 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2016-03-08 15:33:35,648 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2016-03-08 15:33:35,648 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2016-03-08 15:33:35,661 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'}
2016-03-08 15:33:35,662 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0751, 'recursive': True}
2016-03-08 15:33:35,674 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/boot/efi', '/dstage', '/home', '/opt', '/tmp', '/usr', '/var', '/data1', '/data2', '/data3', '/data4', '/data5', '/data6', '/data7', '/data8', '/data9', '/data10', '/proc/sys/fs/binfmt_misc', '/bdata1', '/bdata2', '/bdata3', '/data4/ramdisk'].
2016-03-08 15:33:35,675 - Mount point for directory /bdata1/hadoop/hdfs/data is /
2016-03-08 15:33:35,675 - Mount point for directory /bdata2/hadoop/hdfs/data is /
2016-03-08 15:33:35,675 - Mount point for directory /bdata3/hadoop/hdfs/data is /
2016-03-08 15:33:35,675 - File['/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-03-08 15:33:35,676 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2016-03-08 15:33:35,676 - Changing owner for /var/run/hadoop from 0 to hdfs
2016-03-08 15:33:35,676 - Changing group for /var/run/hadoop from 0 to hadoop
2016-03-08 15:33:35,677 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2016-03-08 15:33:35,677 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2016-03-08 15:33:35,677 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2016-03-08 15:33:35,692 - Deleting File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid']
2016-03-08 15:33:35,692 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}

avatar
Super Guru

Try

ls -l /usr/hdp/current/hadoop-client/conf/

Do you see config files there, like hdfs-site.xml etc? If not something is wrong. If installed by Ambari, /usr/hdp/current/hadoop-client/conf should be symlink to /etc/hadoop/<hdp-version>/0. If all that is fine, go to one DN, and check its logs. HTH.

avatar

I got the same error and config files are there, could you help us to resolve the issue.

ls -l /usr/hdp/current/hadoop-client/conf/ total 196 -rw-r--r-- 1 hdfs hadoop 2069 May 17 16:07 capacity-scheduler.xml -rw-r--r-- 1 hdfs root 1020 May 17 15:43 commons-logging.properties -rw-r--r-- 1 hdfs hadoop 1335 Feb 11 13:15 configuration.xsl -rw-r--r-- 1 root hadoop 1019 May 17 15:18 container-executor.cfg -rw-r--r-- 1 hdfs hadoop 2535 May 17 16:07 core-site.xml -rw-r--r-- 1 root root 415 May 17 16:07 dfs_data_dir_mount.hist -rw-r--r-- 1 hdfs hadoop 1 May 17 15:57 dfs.exclude -rw-r--r-- 1 root root 3979 Feb 11 13:15 hadoop-env.cmd -rw-r--r-- 1 hdfs hadoop 5370 May 17 15:08 hadoop-env.sh -rw-r--r-- 1 hdfs root 899 May 17 15:43 hadoop-metrics2.properties -rw-r--r-- 1 root root 2490 Feb 11 13:15 hadoop-metrics.properties -rw-r--r-- 1 hdfs hadoop 1342 May 17 16:07 hadoop-policy.xml -rw-r--r-- 1 hdfs hadoop 6575 May 17 16:07 hdfs-site.xml -rw-r--r-- 1 hdfs root 1602 May 17 15:43 health_check -rw-r--r-- 1 root root 3518 Feb 11 13:15 kms-acls.xml -rw-r--r-- 1 root root 1527 Feb 11 13:15 kms-env.sh -rw-r--r-- 1 root root 1631 Feb 11 13:15 kms-log4j.properties -rw-r--r-- 1 root root 5511 Feb 11 13:15 kms-site.xml -rw-r--r-- 1 hdfs hadoop 8699 May 17 15:43 log4j.properties -rw-r--r-- 1 root root 951 Feb 11 13:15 mapred-env.cmd -rw-r--r-- 1 hdfs root 661 May 17 15:18 mapred-env.sh -rw-r--r-- 1 root root 4113 Feb 11 13:15 mapred-queues.xml.template -rw-r--r-- 1 mapred hadoop 6571 May 17 16:07 mapred-site.xml -rw-r--r-- 1 root root 758 Feb 11 13:15 mapred-site.xml.template drwxr-xr-x 2 root hadoop 4096 May 17 15:18 secure -rw-r--r-- 1 hdfs root 26 May 17 15:18 slaves -rw-r--r-- 1 hdfs hadoop 918 May 17 16:07 ssl-client.xml -rw-r--r-- 1 mapred hadoop 2316 Feb 11 13:15 ssl-client.xml.example -rw-r--r-- 1 hdfs hadoop 1034 May 17 16:07 ssl-server.xml -rw-r--r-- 1 mapred hadoop 2268 Feb 11 13:15 ssl-server.xml.example -rw-r--r-- 1 hdfs root 945 May 17 15:18 taskcontroller.cfg -rwxr-xr-x 1 root root 4221 May 17 15:43 task-log4j.properties -rw-r--r-- 1 hdfs hadoop 86 May 17 15:43 topology_mappings.data -rwxr-xr-x 1 root root 2358 May 17 15:43 topology_script.py -rw-r--r-- 1 root root 2250 Feb 11 13:15 yarn-env.cmd -rwxr-xr-x 1 yarn hadoop 4908 May 17 16:07 yarn-env.sh -rw-r--r-- 1 yarn hadoop 0 May 17 15:18 yarn.exclude -rw-r--r-- 1 yarn hadoop 14304 May 17 16:07 yarn-site.xml

avatar

Issue is resolved after adding the parameter in

/etc/security/limits.d/hdfs.conf

hdfs - nofile 32768 hdfs - nproc 65536

And then restart the datanode.