Yarn is not starting up after kerberos installation

[#1]

HI All,

After enabling kerberos yarn service not able to start.

[root@ip-192-168-0-50 keytabs]# ll /etc/security/keytabs total 56

-r--------. 1 root root 588 Nov 13 13:50 activity-explorer.headless.keytab -r--r-----. 1 hbase hadoop 383 Nov 13 13:50 hbase.headless.keytab -r--------. 1 hbase hadoop 528 Nov 13 13:50 hbase.service.keytab -r--------. 1 hdfs hadoop 378 Nov 13 13:50 hdfs.headless.keytab -r--------. 1 hue hue 518 Nov 13 13:50 hue.service.keytab -rw-r-----. 1 ambari-qa hadoop 383 Nov 11 11:51 kerberos.service_check.111117.keytab -r--------. 1 storm hadoop 533 Nov 13 13:50 nimbus.service.keytab -r--------. 1 hdfs hadoop 513 Nov 13 13:50 nn.service.keytab -r--------. 1 yarn hadoop 513 Nov 13 13:50 rm.service.keytab -r--r-----. 1 ambari-qa hadoop 403 Nov 13 13:50 smokeuser.headless.keytab -r--------. 1 spark hadoop 383 Nov 13 13:50 spark.headless.keytab -r--r-----. 1 root hadoop 523 Nov 13 13:50 spnego.service.keytab -r--------. 1 storm hadoop 383 Nov 13 13:50 storm.headless.keytab -r--------. 1 zookeeper hadoop 548 Nov 13 13:50 zk.service.keytab [root@ip-192-168-0-50 keytabs]#

Its giving error like below :

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 304, in <module>
    Resourcemanager().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 124, in start
    self.wait_for_dfs_directories_created(params.entity_groupfs_store_dir, params.entity_groupfs_active_dir)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 254, in wait_for_dfs_directories_created
    user=params.yarn_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/rm.service.keytab rm/ip-192-168-0-50.eu-west-1.compute.internal@TECHNIPFMC.COM;' returned 1. kinit: Preauthentication failed while getting initial credentials

2017-11-13 18:23:25,071 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-13 18:23:25,181 - Stack Feature Version Info: stack_version=2.5, version=2.5.5.0-157, current_cluster_version=2.5.5.0-157 -> 2.5.5.0-157
2017-11-13 18:23:25,182 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2017-11-13 18:23:25,183 - Group['livy'] {}
2017-11-13 18:23:25,191 - Group['spark'] {}
2017-11-13 18:23:25,191 - Group['hue'] {}
2017-11-13 18:23:25,191 - Group['hadoop'] {}
2017-11-13 18:23:25,192 - Group['users'] {}
2017-11-13 18:23:25,192 - Group['knox'] {}
2017-11-13 18:23:25,192 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,193 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,194 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,194 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,195 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-11-13 18:23:25,195 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,196 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,197 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-11-13 18:23:25,197 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,198 - User['hue'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,198 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,199 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,200 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,200 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,201 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,202 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-11-13 18:23:25,202 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-13 18:23:25,204 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-11-13 18:23:25,208 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-11-13 18:23:25,208 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2017-11-13 18:23:25,211 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-13 18:23:25,212 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2017-11-13 18:23:25,216 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2017-11-13 18:23:25,216 - Group['hdfs'] {}
2017-11-13 18:23:25,216 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2017-11-13 18:23:25,217 - FS Type: 
2017-11-13 18:23:25,217 - Directory['/etc/hadoop'] {'mode': 0755}
2017-11-13 18:23:25,230 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2017-11-13 18:23:25,230 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2017-11-13 18:23:25,242 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2017-11-13 18:23:25,254 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2017-11-13 18:23:25,255 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2017-11-13 18:23:25,256 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2017-11-13 18:23:25,259 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2017-11-13 18:23:25,261 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2017-11-13 18:23:25,267 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-11-13 18:23:25,281 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2017-11-13 18:23:25,282 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2017-11-13 18:23:25,283 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2017-11-13 18:23:25,286 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2017-11-13 18:23:25,289 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2017-11-13 18:23:25,292 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
2017-11-13 18:23:25,293 - Execute['/usr/jdk64/jdk1.8.0_112/bin/java -jar /var/lib/ambari-agent/tools/jcepolicyinfo.jar -tu'] {'logoutput': True, 'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_112'}}
Unlimited Key JCE Policy: true
2017-11-13 18:23:25,473 - The unlimited key JCE policy is required, and appears to have been installed.
2017-11-13 18:23:25,685 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-13 18:23:25,686 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager'] {'timeout': 20}
2017-11-13 18:23:25,705 - call returned (0, 'hadoop-yarn-resourcemanager - 2.5.5.0-157')
2017-11-13 18:23:25,706 - Stack Feature Version Info: stack_version=2.5, version=2.5.5.0-157, current_cluster_version=2.5.5.0-157 -> 2.5.5.0-157
2017-11-13 18:23:25,708 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-13 18:23:25,716 - Directory['/var/log/hadoop-yarn/nodemanager/recovery-state'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2017-11-13 18:23:25,718 - Directory['/var/run/hadoop-yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-11-13 18:23:25,718 - Directory['/var/run/hadoop-yarn/yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-11-13 18:23:25,718 - Directory['/var/log/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2017-11-13 18:23:25,719 - Directory['/var/run/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-11-13 18:23:25,719 - Directory['/var/run/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-11-13 18:23:25,720 - Directory['/var/log/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-11-13 18:23:25,720 - Directory['/var/log/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2017-11-13 18:23:25,721 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn', 'group': 'hadoop', 'ignore_failures': True, 'create_parents': True, 'cd_access': 'a'}
2017-11-13 18:23:25,721 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'fs.defaultFS': 'true'}}, 'owner': 'hdfs', 'configurations': ...}
2017-11-13 18:23:25,728 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2017-11-13 18:23:25,729 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-11-13 18:23:25,755 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'dfs.support.append': 'true', 'dfs.datanode.data.dir': 'true', 'dfs.namenode.http-address': 'true', 'dfs.namenode.name.dir': 'true', 'dfs.webhdfs.enabled': 'true', 'dfs.datanode.failed.volumes.tolerated': 'true'}}, 'owner': 'hdfs', 'configurations': ...}
2017-11-13 18:23:25,762 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2017-11-13 18:23:25,762 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-11-13 18:23:25,806 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2017-11-13 18:23:25,813 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2017-11-13 18:23:25,813 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-11-13 18:23:25,848 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 513 to yarn
2017-11-13 18:23:25,848 - XmlConfig['yarn-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2017-11-13 18:23:25,855 - Generating config: /usr/hdp/current/hadoop-client/conf/yarn-site.xml
2017-11-13 18:23:25,855 - File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-11-13 18:23:25,946 - XmlConfig['capacity-scheduler.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2017-11-13 18:23:25,953 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2017-11-13 18:23:25,953 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-11-13 18:23:25,965 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 510 to yarn
2017-11-13 18:23:25,965 - Directory['/etc/hadoop/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2017-11-13 18:23:25,965 - File['/etc/hadoop/conf/yarn.exclude'] {'owner': 'yarn', 'group': 'hadoop'}
2017-11-13 18:23:25,966 - File['/var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log'] {'owner': 'yarn', 'group': 'hadoop'}
2017-11-13 18:23:25,968 - File['/etc/security/limits.d/yarn.conf'] {'content': Template('yarn.conf.j2'), 'mode': 0644}
2017-11-13 18:23:25,970 - File['/etc/security/limits.d/mapreduce.conf'] {'content': Template('mapreduce.conf.j2'), 'mode': 0644}
2017-11-13 18:23:25,974 - File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] {'content': InlineTemplate(...), 'owner': 'yarn', 'group': 'hadoop', 'mode': 0755}
2017-11-13 18:23:25,975 - Writing File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] because contents don't match
2017-11-13 18:23:25,975 - File['/usr/hdp/current/hadoop-yarn-resourcemanager/bin/container-executor'] {'group': 'hadoop', 'mode': 06050}
2017-11-13 18:23:25,977 - File['/usr/hdp/current/hadoop-client/conf/container-executor.cfg'] {'content': Template('container-executor.cfg.j2'), 'group': 'hadoop', 'mode': 0644}
2017-11-13 18:23:25,977 - Directory['/cgroups_test/cpu'] {'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2017-11-13 18:23:25,979 - File['/usr/hdp/current/hadoop-client/conf/mapred-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'mode': 0755}
2017-11-13 18:23:25,980 - File['/usr/hdp/current/hadoop-client/sbin/task-controller'] {'owner': 'root', 'group': 'hadoop', 'mode': 06050}
2017-11-13 18:23:25,982 - File['/usr/hdp/current/hadoop-client/conf/taskcontroller.cfg'] {'content': Template('taskcontroller.cfg.j2'), 'owner': 'root', 'group': 'hadoop', 'mode': 0644}
2017-11-13 18:23:25,983 - File['/usr/hdp/current/hadoop-client/conf/yarn_jaas.conf'] {'content': Template('yarn_jaas.conf.j2'), 'owner': 'yarn', 'group': 'hadoop'}
2017-11-13 18:23:25,984 - XmlConfig['mapred-site.xml'] {'owner': 'mapred', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2017-11-13 18:23:25,990 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2017-11-13 18:23:25,991 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'mapred', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-11-13 18:23:26,025 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 512 to mapred
2017-11-13 18:23:26,025 - XmlConfig['capacity-scheduler.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2017-11-13 18:23:26,032 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2017-11-13 18:23:26,032 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-11-13 18:23:26,044 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 512 to hdfs
2017-11-13 18:23:26,044 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2017-11-13 18:23:26,051 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2017-11-13 18:23:26,051 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-11-13 18:23:26,056 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-11-13 18:23:26,057 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2017-11-13 18:23:26,063 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2017-11-13 18:23:26,064 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-11-13 18:23:26,069 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2017-11-13 18:23:26,075 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2017-11-13 18:23:26,076 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2017-11-13 18:23:26,081 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2017-11-13 18:23:26,082 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2017-11-13 18:23:26,082 - Verifying DFS directories where ATS stores time line data for active and completed applications.
2017-11-13 18:23:26,082 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/rm.service.keytab rm/ip-192-168-0-50.eu-west-1.compute.internal@TECHNIPFMC.COM;'] {'user': 'yarn'}

Command failed after 1 tries

[#2]

Hi @Priyaranjan Swain

Try running following command on host where you have yarn resource Manager "RM" installed.

/usr/bin/kinit -kt /etc/security/keytabs/rm.service.keytab rm/ip-192-168-0-50.eu-west-1.compute.internal@TECHNIPFMC.COM;

If this does not return the following error, then try restarting service again via Ambari.

returned 1. kinit: Preauthentication failed while getting initial credentials

If returns above error, check the /etc/krb5.conf to check for any issues here. or in case of KDC Load balancer, see if the principals are replicated across all servers serving under this KDC.

[realms]
  TECHNIPFMC.COM = {
    admin_server = <FQDN>
    kdc = <FQDN>
  }

[#3]

HI Saumil,

After runnning this command am getting below error .

1. /usr/bin/kinit -kt /etc/security/keytabs/rm.service.keytab rm/ip-192-168-0-50.eu-west-1.compute.internal@TECHNIPFMC.COM;

Same error:

/usr/bin/kinit -kt /etc/security/keytabs/rm.service.keytab rm/ip-192-168-0-50.eu-west-1.compute.internal@TECHNIPFMC.COM; kinit: Preauthentication failed while getting initial credentials

Run both as root user and yarn user also .. both same error.

[#4]

Hi @Priyaranjan Swain

Is this Kerberos enabled via Existing MIT KDC or Existing Active Directory?

Please check if the principal is locked in your MIT KDC or Active Directory. Also as mention earlier, please check the /etc/krb5.conf for any issues in this file with respect to KDC information. You may try "Regenerating Keytabs" via Ambari-server and see if it helps.

[#5]

Hi @Priyaranjan Swain

Is this Kerberos enabled via Existing MIT KDC or Existing Active Directory?

Please check if the principal is locked in your MIT KDC or Active Directory. Also as mention earlier, please check the /etc/krb5.conf for any issues in this file with respect to KDC information. You may try "Regenerating Keytabs" via Ambari-server and see if it helps.