Hiveserver interactive(LLAP) startup failure javax.security.auth.login.LoginException: Pre-authentication information was invalid (24)

[#1]

LLAP Service sometimes may fail to startup after HDP upgrade with the following error: 

Error::

Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hive/prakash.cloudera.com@CLOUDERA.COM from keytab /grid/0/yarn/local/usercache/hive/appcache/application_**********/container_***********_01_000001/keytabs/llap0.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24)

To resolve this issue, run the following command and start LLAP: 

yarn app -destroy llap0

If the issue persists, then do the following

1. Check the hive.service.keytab file is available in all the hosts, run kinit and test the connectivity.
2. If the file is missing, recreate the keytabs on the missing hosts either through Ambari or manually (see this article for manual keytab creation).
3. Verify the file in HDFS and copy it to the local. Perform kinit with the copied keytab.

   hdfs:///user/hive/.yarn/keytabs/hive/hive.service.keytab

   hdfs dfs -copyToLocal /user/hive/.yarn/keytabs/hive/hive.service.keytab /tmp/hive.keytab
   
   

   kinit -kt /tmp/hive.keytab hive/prakash.cloudera.com@CLOUDERA.COM
   kinit: Preauthentication failed while getting initial credentials

4.  If you get the error, to move the keytab file from hdfs location.

   hdfs dfs -cp /user/hive/.yarn/keytabs/hive/hive.service.keytab /tmp/
   hdfs dfs -rm /user/hive/.yarn/keytabs/hive/hive.service.keytab

5. Start LLAP component.

[#2]

@Folks You can follow this approach for any other kerberos keytab related issues with LLAP.

[#3]

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/ambari-release-notes/content/known_issues.html