Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Standby Namenode fails to start after kerberization of HDP Cluster with following error

Highlighted

Standby Namenode fails to start after kerberization of HDP Cluster with following error

New Contributor

Here is hadoop-hdfs-zkfc-standbynamenode.log file

2017-08-31 08:16:12,503 WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(211)) - Transport-level exception trying to monitor health of NameNode at standbynamenode-d604uwrvdk.local/10.0.0.250:8020: java.net.ConnectException: Connection refused Call From standbynamenode-d604uwrvdk.local/10.0.0.250 to standbynamenode-d604uwrvdk.local:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
2017-08-31 08:16:14,505 INFO ipc.Client (Client.java:handleConnectionFailure(903)) - Retrying connect to server: standbynamenode-d604uwrvdk.local/10.0.0.250:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
2017-08-31 08:16:14,506 WARN ipc.Client (Client.java:handleConnectionFailure(885)) - Failed to connect to server: standbynamenode-d604uwrvdk.local/10.0.0.250:8020: retries get failed due to exceeded maximum allowed retries number: 1
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:649)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:744)
at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1431)
at org.apache.hadoop.ipc.Client.call(Client.java:1392)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:122)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:297)

7 REPLIES 7

Re: Standby Namenode fails to start after kerberization of HDP Cluster with following error

New Contributor

Hi @Mahesh Thumar,

You have actually given the zkfc logs not the namenode logs. This zkfc logs suggests that namenode is not running.
Please look into namenode's log for actual error information regarding the namenode startup failure.

Re: Standby Namenode fails to start after kerberization of HDP Cluster with following error

New Contributor

Re: Standby Namenode fails to start after kerberization of HDP Cluster with following error

Hi @Mahesh Thumar,

As per the logs it looks like a FQDN resolution issue to me .Please check if your fqdn/hostname matches with the hostnames given in ambari.

Thanks,

Aditya

Re: Standby Namenode fails to start after kerberization of HDP Cluster with following error

Mentor

@Mahesh Thumar

The below error clearly shows there is a mismatch between you principal created in KDC and the hostname in /etc/hosts

Whats the output of

cat /etc/hosts

Can you validate the namenode host name linked to the principal by connecting to kadmin

# kadmin.local 
Authenticating as principal root/admin@HDP.LOCAL with password.
kadmin.local: listprincs nn*

The output should match after nn/xxx should match the FQDN in the /etc/hosts remove the ALIAS standbynamenode-t5liwl46t1.hdp.local if it exists in your /etc/hosts

IP:FQDN:ALIAS
# In the below example I removed the alias
192.168.10.234     ec2-34-208-160-33.us-west-2.compute.amazonaws.com  

8485: User nn/standbynamenode-t5liwl46t1.hdp.local@HDP.LOCAL (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol, expected client Kerberos principal is nn/ec2-34-208-160-33.us-west-2.compute.amazonaws.com@HDP.LOCAL

You may need to restart the network services

Hope that helps


Re: Standby Namenode fails to start after kerberization of HDP Cluster with following error

New Contributor

hadoop-hdfs-namenode-standbynamenode.txt

Hi @Geoffrey Shelton Okot @Aditya Sirna

Thank you for your support for solving the FQDN issue.

FQDN issue is resolved, but still i'm not able to start my namenode on standbynamenode. I have also attached log file for standby namenode.

Stderr:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 408, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 103, in start
    upgrade_suspended=params.upgrade_suspended, env=env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 118, in namenode
    raise Fail("Could not bootstrap standby namenode")
resource_management.core.exceptions.Fail: Could not bootstrap standby namenode

stdout:

2017-09-29 11:54:57,546 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.3.0-227
2017-09-29 11:54:57,547 - Checking if need to create versioned conf dir /etc/hadoop/2.4.3.0-227/0
2017-09-29 11:54:57,547 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-09-29 11:54:57,593 - call returned (1, '/etc/hadoop/2.4.3.0-227/0 exist already', '')
2017-09-29 11:54:57,593 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-09-29 11:54:57,610 - checked_call returned (0, '')
2017-09-29 11:54:57,610 - Ensuring that hadoop has the correct symlink structure
2017-09-29 11:54:57,622 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-09-29 11:54:57,806 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.3.0-227
2017-09-29 11:54:57,806 - Checking if need to create versioned conf dir /etc/hadoop/2.4.3.0-227/0
2017-09-29 11:54:57,807 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-09-29 11:54:57,825 - call returned (1, '/etc/hadoop/2.4.3.0-227/0 exist already', '')
2017-09-29 11:54:57,825 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-09-29 11:54:57,842 - checked_call returned (0, '')
2017-09-29 11:54:57,842 - Ensuring that hadoop has the correct symlink structure
2017-09-29 11:54:57,842 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-09-29 11:54:57,843 - Group['spark'] {}
2017-09-29 11:54:57,876 - Group['ranger'] {}
2017-09-29 11:54:57,876 - Group['hadoop'] {}
2017-09-29 11:54:57,877 - Group['users'] {}
2017-09-29 11:54:57,877 - Group['knox'] {}
2017-09-29 11:54:57,877 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,878 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,878 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2017-09-29 11:54:57,879 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,880 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2017-09-29 11:54:57,880 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2017-09-29 11:54:57,881 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2017-09-29 11:54:57,882 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,882 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2017-09-29 11:54:57,883 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,883 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,884 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,885 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,885 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,886 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,887 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,887 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-09-29 11:54:57,888 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-09-29 11:54:57,902 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-09-29 11:54:57,906 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-09-29 11:54:57,906 - Group['hdfs'] {}
2017-09-29 11:54:57,907 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2017-09-29 11:54:57,907 - FS Type: 
2017-09-29 11:54:57,907 - Directory['/etc/hadoop'] {'mode': 0755}
2017-09-29 11:54:57,917 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2017-09-29 11:54:57,919 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2017-09-29 11:54:57,928 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2017-09-29 11:54:57,958 - Skipping Execute[('setenforce', '0')] due to only_if
2017-09-29 11:54:57,959 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2017-09-29 11:54:57,961 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2017-09-29 11:54:57,961 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2017-09-29 11:54:57,966 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2017-09-29 11:54:57,968 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2017-09-29 11:54:57,991 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-09-29 11:54:58,002 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2017-09-29 11:54:58,014 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2017-09-29 11:54:58,028 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2017-09-29 11:54:58,032 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2017-09-29 11:54:58,035 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2017-09-29 11:54:58,224 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.3.0-227
2017-09-29 11:54:58,225 - Checking if need to create versioned conf dir /etc/hadoop/2.4.3.0-227/0
2017-09-29 11:54:58,225 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-09-29 11:54:58,243 - call returned (1, '/etc/hadoop/2.4.3.0-227/0 exist already', '')
2017-09-29 11:54:58,243 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-09-29 11:54:58,261 - checked_call returned (0, '')
2017-09-29 11:54:58,261 - Ensuring that hadoop has the correct symlink structure
2017-09-29 11:54:58,261 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-09-29 11:54:58,277 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.3.0-227
2017-09-29 11:54:58,277 - Checking if need to create versioned conf dir /etc/hadoop/2.4.3.0-227/0
2017-09-29 11:54:58,277 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-09-29 11:54:58,295 - call returned (1, '/etc/hadoop/2.4.3.0-227/0 exist already', '')
2017-09-29 11:54:58,295 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.3.0-227 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-09-29 11:54:58,312 - checked_call returned (0, '')
2017-09-29 11:54:58,312 - Ensuring that hadoop has the correct symlink structure
2017-09-29 11:54:58,313 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-09-29 11:54:58,319 - Directory['/etc/security/limits.d'] {'owner': 'root', 'group': 'root', 'recursive': True}
2017-09-29 11:54:58,353 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2017-09-29 11:54:58,354 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2017-09-29 11:54:58,361 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2017-09-29 11:54:58,361 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,369 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2017-09-29 11:54:58,376 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2017-09-29 11:54:58,376 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,381 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2017-09-29 11:54:58,382 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2017-09-29 11:54:58,388 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2017-09-29 11:54:58,388 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,396 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2017-09-29 11:54:58,402 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2017-09-29 11:54:58,402 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,409 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {u'final': {u'dfs.datanode.failed.volumes.tolerated': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.support.append': u'true', u'dfs.webhdfs.enabled': u'true'}}, 'configurations': ...}
2017-09-29 11:54:58,416 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2017-09-29 11:54:58,416 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,465 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2017-09-29 11:54:58,471 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2017-09-29 11:54:58,471 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,498 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'root'}
2017-09-29 11:54:58,499 - Directory['/data/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'recursive': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'}
2017-09-29 11:54:58,532 - Called service start with upgrade_type: None
2017-09-29 11:54:58,560 - HDFS: Setup ranger: command retry not enabled thus skipping if ranger admin is down !
2017-09-29 11:54:58,580 - Rangeradmin: Skip ranger admin if it's down !
2017-09-29 11:54:58,796 - amb_ranger_admin user already exists.
2017-09-29 11:54:58,826 - Hdfs Repository Cluster1_hadoop exist
2017-09-29 11:54:58,827 - File['/usr/hdp/current/hadoop-client/conf/ranger-security.xml'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-09-29 11:54:58,828 - Writing File['/usr/hdp/current/hadoop-client/conf/ranger-security.xml'] because contents don't match
2017-09-29 11:54:58,829 - Directory['/etc/ranger/Cluster1_hadoop'] {'owner': 'hdfs', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0775}
2017-09-29 11:54:58,829 - Directory['/etc/ranger/Cluster1_hadoop/policycache'] {'owner': 'hdfs', 'recursive': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2017-09-29 11:54:58,830 - File['/etc/ranger/Cluster1_hadoop/policycache/hdfs_Cluster1_hadoop.json'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-09-29 11:54:58,830 - XmlConfig['ranger-hdfs-audit.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0744, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2017-09-29 11:54:58,837 - Generating config: /usr/hdp/current/hadoop-client/conf/ranger-hdfs-audit.xml
2017-09-29 11:54:58,837 - File['/usr/hdp/current/hadoop-client/conf/ranger-hdfs-audit.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0744, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,849 - XmlConfig['ranger-hdfs-security.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0744, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2017-09-29 11:54:58,855 - Generating config: /usr/hdp/current/hadoop-client/conf/ranger-hdfs-security.xml
2017-09-29 11:54:58,855 - File['/usr/hdp/current/hadoop-client/conf/ranger-hdfs-security.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0744, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,861 - XmlConfig['ranger-policymgr-ssl.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0744, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2017-09-29 11:54:58,868 - Generating config: /usr/hdp/current/hadoop-client/conf/ranger-policymgr-ssl.xml
2017-09-29 11:54:58,868 - File['/usr/hdp/current/hadoop-client/conf/ranger-policymgr-ssl.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0744, 'encoding': 'UTF-8'}
2017-09-29 11:54:58,874 - Execute[('/usr/hdp/2.4.3.0-227/ranger-hdfs-plugin/ranger_credential_helper.py', '-l', '/usr/hdp/2.4.3.0-227/ranger-hdfs-plugin/install/lib/*', '-f', '/etc/ranger/Cluster1_hadoop/cred.jceks', '-k', 'sslKeyStore', '-v', [PROTECTED], '-c', '1')] {'logoutput': True, 'environment': {'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_60'}, 'sudo': True}
Using Java:/usr/jdk64/jdk1.8.0_60/bin/java
Alias sslKeyStore created successfully!
2017-09-29 11:55:00,624 - Execute[('/usr/hdp/2.4.3.0-227/ranger-hdfs-plugin/ranger_credential_helper.py', '-l', '/usr/hdp/2.4.3.0-227/ranger-hdfs-plugin/install/lib/*', '-f', '/etc/ranger/Cluster1_hadoop/cred.jceks', '-k', 'sslTrustStore', '-v', [PROTECTED], '-c', '1')] {'logoutput': True, 'environment': {'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_60'}, 'sudo': True}
Using Java:/usr/jdk64/jdk1.8.0_60/bin/java
Alias sslTrustStore created successfully!
2017-09-29 11:55:01,599 - File['/etc/ranger/Cluster1_hadoop/cred.jceks'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0640}
2017-09-29 11:55:01,610 - File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'), 'group': 'hadoop'}
2017-09-29 11:55:01,611 - Boostrapping standby namenode: hdfs namenode -bootstrapStandby -nonInteractive
2017-09-29 11:55:01,611 - Try 1 out of 50
2017-09-29 11:55:01,611 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:01,759 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid17769.log")
2017-09-29 11:55:01,760 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:01,760 - Try 2 out of 50
2017-09-29 11:55:01,760 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:01,823 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid17828.log")
2017-09-29 11:55:01,823 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:01,823 - Try 3 out of 50
2017-09-29 11:55:01,823 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:01,885 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid17888.log")
2017-09-29 11:55:01,885 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:01,885 - Try 4 out of 50
2017-09-29 11:55:01,885 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:01,947 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid17947.log")
2017-09-29 11:55:01,948 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:01,948 - Try 5 out of 50
2017-09-29 11:55:01,948 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,011 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18006.log")
2017-09-29 11:55:02,011 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,011 - Try 6 out of 50
2017-09-29 11:55:02,011 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,072 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18065.log")
2017-09-29 11:55:02,072 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,073 - Try 7 out of 50
2017-09-29 11:55:02,073 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,134 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18130.log")
2017-09-29 11:55:02,134 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,134 - Try 8 out of 50
2017-09-29 11:55:02,134 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,195 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18197.log")
2017-09-29 11:55:02,195 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,195 - Try 9 out of 50
2017-09-29 11:55:02,195 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,256 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18256.log")
2017-09-29 11:55:02,257 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,257 - Try 10 out of 50
2017-09-29 11:55:02,257 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,318 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18319.log")
2017-09-29 11:55:02,318 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,318 - Try 11 out of 50
2017-09-29 11:55:02,319 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,380 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18378.log")
2017-09-29 11:55:02,380 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,380 - Try 12 out of 50
2017-09-29 11:55:02,380 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,441 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18437.log")
2017-09-29 11:55:02,441 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,441 - Try 13 out of 50
2017-09-29 11:55:02,441 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,503 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18496.log")
2017-09-29 11:55:02,504 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,504 - Try 14 out of 50
2017-09-29 11:55:02,504 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,564 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18555.log")
2017-09-29 11:55:02,564 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,565 - Try 15 out of 50
2017-09-29 11:55:02,565 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,625 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18614.log")
2017-09-29 11:55:02,625 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,626 - Try 16 out of 50
2017-09-29 11:55:02,626 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,686 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18673.log")
2017-09-29 11:55:02,686 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,686 - Try 17 out of 50
2017-09-29 11:55:02,686 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,747 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18732.log")
2017-09-29 11:55:02,747 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,747 - Try 18 out of 50
2017-09-29 11:55:02,747 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,808 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18791.log")
2017-09-29 11:55:02,808 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,808 - Try 19 out of 50
2017-09-29 11:55:02,809 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,869 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18850.log")
2017-09-29 11:55:02,869 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,869 - Try 20 out of 50
2017-09-29 11:55:02,869 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,930 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18909.log")
2017-09-29 11:55:02,930 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,930 - Try 21 out of 50
2017-09-29 11:55:02,930 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:02,991 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid18968.log")
2017-09-29 11:55:02,992 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:02,992 - Try 22 out of 50
2017-09-29 11:55:02,992 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,053 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19028.log")
2017-09-29 11:55:03,053 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,053 - Try 23 out of 50
2017-09-29 11:55:03,053 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,114 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19087.log")
2017-09-29 11:55:03,114 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,114 - Try 24 out of 50
2017-09-29 11:55:03,114 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,175 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19146.log")
2017-09-29 11:55:03,175 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,175 - Try 25 out of 50
2017-09-29 11:55:03,175 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,238 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19205.log")
2017-09-29 11:55:03,238 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,238 - Try 26 out of 50
2017-09-29 11:55:03,238 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,300 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19264.log")
2017-09-29 11:55:03,301 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,301 - Try 27 out of 50
2017-09-29 11:55:03,301 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,363 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19324.log")
2017-09-29 11:55:03,363 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,363 - Try 28 out of 50
2017-09-29 11:55:03,363 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,426 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19383.log")
2017-09-29 11:55:03,426 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,426 - Try 29 out of 50
2017-09-29 11:55:03,426 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,490 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19444.log")
2017-09-29 11:55:03,490 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,490 - Try 30 out of 50
2017-09-29 11:55:03,490 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,554 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19503.log")
2017-09-29 11:55:03,554 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,554 - Try 31 out of 50
2017-09-29 11:55:03,555 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,616 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19563.log")
2017-09-29 11:55:03,616 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,616 - Try 32 out of 50
2017-09-29 11:55:03,616 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,677 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19623.log")
2017-09-29 11:55:03,677 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,678 - Try 33 out of 50
2017-09-29 11:55:03,678 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,738 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19682.log")
2017-09-29 11:55:03,739 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,739 - Try 34 out of 50
2017-09-29 11:55:03,739 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,801 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19741.log")
2017-09-29 11:55:03,801 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,801 - Try 35 out of 50
2017-09-29 11:55:03,801 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,862 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19812.log")
2017-09-29 11:55:03,863 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,863 - Try 36 out of 50
2017-09-29 11:55:03,863 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,924 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19871.log")
2017-09-29 11:55:03,924 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,924 - Try 37 out of 50
2017-09-29 11:55:03,924 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:03,987 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid19941.log")
2017-09-29 11:55:03,987 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:03,987 - Try 38 out of 50
2017-09-29 11:55:03,987 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,049 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20000.log")
2017-09-29 11:55:04,049 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,049 - Try 39 out of 50
2017-09-29 11:55:04,049 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,111 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20059.log")
2017-09-29 11:55:04,111 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,111 - Try 40 out of 50
2017-09-29 11:55:04,112 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,173 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20118.log")
2017-09-29 11:55:04,173 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,173 - Try 41 out of 50
2017-09-29 11:55:04,173 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,243 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20178.log")
2017-09-29 11:55:04,243 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,244 - Try 42 out of 50
2017-09-29 11:55:04,244 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,307 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20238.log")
2017-09-29 11:55:04,307 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,307 - Try 43 out of 50
2017-09-29 11:55:04,308 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,368 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20297.log")
2017-09-29 11:55:04,369 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,369 - Try 44 out of 50
2017-09-29 11:55:04,369 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,430 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20357.log")
2017-09-29 11:55:04,430 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,430 - Try 45 out of 50
2017-09-29 11:55:04,430 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,491 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20418.log")
2017-09-29 11:55:04,491 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,491 - Try 46 out of 50
2017-09-29 11:55:04,491 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,552 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20477.log")
2017-09-29 11:55:04,552 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,552 - Try 47 out of 50
2017-09-29 11:55:04,552 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,614 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20536.log")
2017-09-29 11:55:04,614 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,614 - Try 48 out of 50
2017-09-29 11:55:04,614 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,675 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20595.log")
2017-09-29 11:55:04,675 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,675 - Try 49 out of 50
2017-09-29 11:55:04,675 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,779 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20664.log")
2017-09-29 11:55:04,779 - Bootstrap standby namenode failed with 1 error code. Will retry
2017-09-29 11:55:04,779 - Try 50 out of 50
2017-09-29 11:55:04,779 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2017-09-29 11:55:04,933 - call returned (1, "Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid20799.log")
2017-09-29 11:55:04,934 - Bootstrap standby namenode failed with 1 error code. Will retry

Any information regarding this error.

Re: Standby Namenode fails to start after kerberization of HDP Cluster with following error

@Mahesh Thumar,

As per the logs, looks like you have run out of memory. please increase the swap memory and try starting the namenode.

Thanks,

Aditya

Re: Standby Namenode fails to start after kerberization of HDP Cluster with following error

Mentor

@Mahesh Thumar,

Your problem is below , please allocate more memory

2017-09-2911:55:01,759- call returned (1,"Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c800000, 1937768448, 0) failed; error='Cannot allocate memory' (errno=12)\n#\n# There is insufficient memory for the Java Runtime Environment to continue.\n# Native memory allocation (mmap) failed to map 1937768448 bytes for committing reserved memory.\n# An error report file with more information is saved as:\n# /var/log/hadoop/hdfs/hs_err_pid17769.log")