Error while Enabling Kerberos

[#1]

Hello Everyone,

I'm trying to enable kerberos on my HDP setup. during the wizard, it fails at Check Kerberos step. Before that, it shows that all the Keytabs creation and everything is getting successful.

But the keytabs are not actually created. Please check the below screenshot:

I checked in the host in question and the keytab file indeed didn't exist.

Can anyone please give ideas where it might be going wrong?

In the ambari-server log file I see the below error snippet.

27 Jan 2021 18:34:52,512  INFO [Server Action Executor Worker 6943] KerberosServerAction:353 - Processing identities...
27 Jan 2021 18:34:52,730 ERROR [Server Action Executor Worker 6943] FinalizeKerberosServerAction:114 - Failed to update the owner of the keytab file at /etc/security/keytabs/kerberos.service_check.012721.keytab to ambari-qa: chown: cannot access ‘/etc/security/keytabs/kerberos.service_check.012721.keytab’: No such file or directory

27 Jan 2021 18:34:52,738 ERROR [Server Action Executor Worker 6943] FinalizeKerberosServerAction:128 - Failed to update the group of the keytab file at /etc/security/keytabs/kerberos.service_check.012721.keytab to hadoop: chgrp: cannot access ‘/etc/security/keytabs/kerberos.service_check.012721.keytab’: No such file or directory

27 Jan 2021 18:34:52,745 ERROR [Server Action Executor Worker 6943] FinalizeKerberosServerAction:145 - Failed to update the access mode of the keytab file at /etc/security/keytabs/kerberos.service_check.012721.keytab to owner:'rw' and group:'rw': chmod: cannot access ‘/etc/security/keytabs/kerberos.service_check.012721.keytab’: No such file or directory

27 Jan 2021 18:34:52,746  INFO [Server Action Executor Worker 6943] KerberosServerAction:457 - Processing identities completed.
27 Jan 2021 18:34:53,746  INFO [qtp-ambari-agent-322] HeartBeatHandler:332 - HeartBeatHandler.sendCommands: sending ExecutionCommand for host checldbipba02.npci.org.in, role KERBEROS_SERVICE_CHECK, roleCommand SERVICE_CHECK, and command ID 546-5, task ID 6944
27 Jan 2021 18:34:55,522 ERROR [ambari-action-scheduler] ActionScheduler:447 - Operation completely failed, aborting request id: 546

Thanks,

Megh

[#2]

Tried this today once again.

Now the Keytab is generated but it doesn't contain any credentials.

$ sudo -u ambari-qa klist -kte /etc/security/keytabs/kerberos.service_check.012821.keytab
Keytab name: FILE:/etc/security/keytabs/kerberos.service_check.012821.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------

The error log looks like this:

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/KERBEROS/1.10.3-10/package/scripts/service_check.py", line 81, in <module>
    KerberosServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/KERBEROS/1.10.3-10/package/scripts/service_check.py", line 64, in service_check
    user=params.smoke_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/kerberos_service_check_cc_2d8eec0f439ec94bde482d7a8511a9c5 -kt /etc/security/keytabs/kerberos.service_check.012821.keytab bhimdr-012821@<EXAMPLE.COM>' returned 1. kinit: Keytab contains no suitable keys for bhimdr-012821@<EXAMPLE.COM> while getting initial credentials
 stdout:
Performing kinit using bhimdr-012821@<EXAMPLE.COM>
2021-01-28 11:15:14,773 - Execute['/usr/bin/kinit -c /var/lib/ambari-agent/tmp/kerberos_service_check_cc_2d8eec0f439ec94bde482d7a8511a9c5 -kt /etc/security/keytabs/kerberos.service_check.012821.keytab bhimdr-012821@<EXAMPLE.COM>'] {'user': 'ambari-qa'}
2021-01-28 11:15:14,942 - File['/var/lib/ambari-agent/tmp/kerberos_service_check_cc_2d8eec0f439ec94bde482d7a8511a9c5'] {'action': ['delete']}

Command failed after 1 tries

I have replaced my actual Realm name with <EXAMPLE.COM> in the above log.

Thanks,

Megh

[#3]

Figured out the keytab error.

I had changed the encryption types and it was resulting in this error. I reverted those changes in the Kerberos configuration and it came clean.

But after the entire process, now the services are not coming up.

Firstly, it is getting stuck on starting the namenode, with safe mode not getting turned off automatically.

If I leave the namenode manually, then it is showing the below error:

stderr: 
2021-02-01 17:18:32,891 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF.
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 367, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 850, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 100, in start
    upgrade_suspended=params.upgrade_suspended, env=env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 226, in namenode
    create_hdfs_directories()
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 293, in create_hdfs_directories
    mode=0777,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 555, in action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 552, in action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 279, in action_delayed
    self._assert_valid()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 238, in _assert_valid
    self.target_status = self._get_file_status(target)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 381, in _get_file_status
    list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u : 'http://<namenode-hostname>:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'' returned status_code=403. 
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 org.apache.hadoop.security.authentication.client.AuthenticationException</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/tmp. Reason:
<pre>    org.apache.hadoop.security.authentication.client.AuthenticationException</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                

</body>
</html>
 stdout:
2021-02-01 16:55:36,478 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2021-02-01 16:55:36,639 - Stack Feature Version Info: stack_version=2.6, version=2.6.1.0-129, current_cluster_version=2.6.1.0-129 -> 2.6.1.0-129
2021-02-01 16:55:36,645 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2021-02-01 16:55:36,646 - Group['kms'] {}
2021-02-01 16:55:36,648 - Group['livy'] {}
2021-02-01 16:55:36,648 - Group['spark'] {}
2021-02-01 16:55:36,648 - Group['ranger'] {}
2021-02-01 16:55:36,648 - Group['hadoop'] {}
2021-02-01 16:55:36,648 - Group['users'] {}
2021-02-01 16:55:36,648 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,649 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,650 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,650 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,651 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,651 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2021-02-01 16:55:36,652 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,652 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,653 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,653 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,654 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,654 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,655 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,655 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,656 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,656 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,657 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,657 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2021-02-01 16:55:36,659 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2021-02-01 16:55:36,667 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2021-02-01 16:55:36,667 - Group['hdfs'] {}
2021-02-01 16:55:36,667 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2021-02-01 16:55:36,668 - FS Type: 
2021-02-01 16:55:36,668 - Directory['/etc/hadoop'] {'mode': 0755}
2021-02-01 16:55:36,679 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2021-02-01 16:55:36,680 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2021-02-01 16:55:36,696 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2021-02-01 16:55:36,706 - Skipping Execute[('setenforce', '0')] due to not_if
2021-02-01 16:55:36,707 - Directory['/hadoop/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2021-02-01 16:55:36,709 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2021-02-01 16:55:36,709 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2021-02-01 16:55:36,713 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2021-02-01 16:55:36,714 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2021-02-01 16:55:36,718 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2021-02-01 16:55:36,726 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2021-02-01 16:55:36,726 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2021-02-01 16:55:36,727 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2021-02-01 16:55:36,730 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2021-02-01 16:55:36,736 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2021-02-01 16:55:36,991 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2021-02-01 16:55:36,992 - Stack Feature Version Info: stack_version=2.6, version=2.6.1.0-129, current_cluster_version=2.6.1.0-129 -> 2.6.1.0-129
2021-02-01 16:55:37,011 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2021-02-01 16:55:37,023 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1}
2021-02-01 16:55:37,063 - checked_call returned (0, '2.6.1.0-129', '')
2021-02-01 16:55:37,071 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf stop namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'only_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2021-02-01 16:55:42,273 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete']}
2021-02-01 16:55:42,274 - Pid file /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid is empty or does not exist
2021-02-01 16:55:42,277 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2021-02-01 16:55:42,281 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2021-02-01 16:55:42,281 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,288 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2021-02-01 16:55:42,288 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,295 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,301 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2021-02-01 16:55:42,301 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,306 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2021-02-01 16:55:42,306 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,312 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2021-02-01 16:55:42,313 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,317 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,323 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2021-02-01 16:55:42,323 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,328 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'configurations': ...}
2021-02-01 16:55:42,334 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2021-02-01 16:55:42,334 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,372 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2021-02-01 16:55:42,378 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2021-02-01 16:55:42,378 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,399 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'root'}
2021-02-01 16:55:42,403 - Directory['/hadoop/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2021-02-01 16:55:42,403 - Skipping setting up secure ZNode ACL for HFDS as it's supported only for NameNode HA mode.
2021-02-01 16:55:42,406 - Called service start with upgrade_type: None
2021-02-01 16:55:42,406 - Ranger Hdfs plugin is not enabled
2021-02-01 16:55:42,407 - File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'), 'group': 'hadoop'}
2021-02-01 16:55:42,407 - /hadoop/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted
2021-02-01 16:55:42,407 - Directory['/hadoop/hadoop/hdfs/namenode/namenode-formatted/'] {'create_parents': True}
2021-02-01 16:55:42,407 - Options for start command are: 
2021-02-01 16:55:42,408 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2021-02-01 16:55:42,408 - Changing owner for /var/run/hadoop from 0 to hdfs
2021-02-01 16:55:42,408 - Changing group for /var/run/hadoop from 0 to hadoop
2021-02-01 16:55:42,408 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2021-02-01 16:55:42,408 - Directory['/hadoop/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2021-02-01 16:55:42,409 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2021-02-01 16:55:42,417 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2021-02-01 16:55:46,632 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-<clustername>@<REALM>'] {'user': 'hdfs'}
2021-02-01 16:55:46,843 - Waiting for this NameNode to leave Safemode due to the following conditions: HA: False, isActive: True, upgradeType: None
2021-02-01 16:55:46,844 - Waiting up to 19 minutes for the NameNode to leave Safemode...
2021-02-01 16:55:46,844 - Execute['/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF''] {'logoutput': True, 'tries': 115, 'user': 'hdfs', 'try_sleep': 10}
safemode: Call From <namenode-hostname>/<namenode-ip> to <namenode-hostname>:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2021-02-01 16:55:48,554 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From <namenode-hostname>/<namenode-ip> to <namenode-hostname>:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2021-02-01 16:56:00,609 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:56:12,594 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:56:24,589 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:56:36,590 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:56:48,592 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:57:00,622 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:57:12,563 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:57:24,555 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:57:36,546 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:57:48,553 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:58:00,554 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:58:12,533 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:58:24,552 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:58:36,530 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:58:48,501 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:59:00,476 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:59:12,428 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:59:24,440 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:59:36,419 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 16:59:48,415 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:00:00,402 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:00:12,355 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:00:24,370 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:00:36,342 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:00:48,326 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:01:00,285 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:01:12,241 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:01:24,250 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:01:36,207 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:01:48,155 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:02:00,095 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:02:12,064 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:02:24,063 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:02:36,028 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:02:48,010 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:03:00,007 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:03:11,977 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:03:23,991 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:03:35,968 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:03:47,906 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:03:59,916 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:04:11,873 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:04:23,858 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:04:35,835 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:04:47,801 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:04:59,781 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:05:11,743 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:05:23,700 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:05:35,648 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:05:47,569 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:05:59,518 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:06:11,471 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:06:23,416 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:06:35,407 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:06:47,402 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:06:59,419 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:07:11,393 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:07:23,354 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:07:35,310 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:07:47,282 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:07:59,247 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:08:11,204 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:08:23,163 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:08:35,118 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:08:47,103 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:08:59,117 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:09:11,084 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:09:23,083 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:09:35,067 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:09:47,042 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:09:59,028 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:10:10,987 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:10:22,925 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:10:34,876 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:10:46,824 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:10:58,815 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:11:10,771 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:11:22,711 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:11:34,670 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:11:46,624 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:11:58,620 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:12:10,573 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:12:22,509 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:12:34,444 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:12:46,402 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:12:58,363 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:13:10,327 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:13:22,258 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:13:34,255 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:13:46,189 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:13:58,155 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:14:10,092 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:14:22,046 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:14:33,992 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:14:45,939 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:14:57,904 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:15:09,874 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:15:21,829 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:15:33,784 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:15:45,706 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:15:57,641 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:16:09,595 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:16:21,527 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:16:33,455 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:16:45,379 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:16:57,332 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:17:09,267 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:17:21,205 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:17:33,148 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:17:45,090 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:17:57,066 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:18:08,997 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:18:20,953 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2021-02-01 17:18:32,891 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF.
2021-02-01 17:18:32,891 - HdfsResource['/tmp'] {'security_enabled': True, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'dfs_type': '', 'default_fs': 'hdfs://<namenode-hostname>:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs-<clustername>@<REALM>', 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0777}
2021-02-01 17:18:32,893 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-<clustername>@<REALM>'] {'user': 'hdfs'}
2021-02-01 17:18:33,103 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u : '"'"'http://<namenode-hostname>:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp3JAW_l 2>/tmp/tmpksEFBR''] {'logoutput': None, 'quiet': False}
2021-02-01 17:18:33,291 - call returned (0, '')

Command failed after 1 tries

Anybody can make sense of this?

Thanks,

Megh

[#4 · Accepted solution]

Fixed this by changing my hostnames to lowercase.

Earlier the hostnames were mixed case and researching online I found that Kerberos expects the hostnames to be lowercase and Realm name to be uppercase.

changing the hostnames to all lowercase resolved the issue.

Thanks,

Megh