We are looking into setting up HDFS Snap Shots for our 5.4 cluster. Unfortunately when going into Cloudera Manager (CM) - HDFS - File Browser, we are getting the following errors in our DEV and TEST environment. PROD works fine surprisingly enough.
While the error messages are slightly different, in both cases the keytab files referenced do not exist when we look for them. Anything you can advise us to look at further to try and figure out these errors?
We thought about regenerating the hdfs/ourdevhost.ourcompany.com@OURCOMPANY.COM credentials (CM – Administration – Kerberos – Credentials) but thought we'd ask before doing so. We can't remember the exact details, but I think we've had some issues with this in the past where when regenerating it fails out when trying to create the Active Directory account again because it already exists. We could be off on this so it's why we wanted to check and get a second opinion.
- DEV: Keytab file does not exist.
java.io.IOException: Login failure for hdfs/ourdevhost.ourcompany.com@OURCOMPANY.COM from keytab /tmp/142360606-0/hadoop7884985294202953.keytab Reload root directory.
- TEST: Keytab file does not exist.
java.io.IOException: Failed on local exception: java.io.IOException: Login failure for hdfs/ourtesthost.ourcompany.com@OURCOMPANY.COM from keytab /tmp/144736658-0/hadoop5666268298612442343.keytab; Host Details : local host is: "ourtesthost.ourcompany.com/10.10.104.167"; destination host is: "ourtesthost.ourcompany.com":8020
The short answer here is to restart Cloudera Manager with "service cloudera-scm-server restart"
In Cloudera Manager 5.4, the keytabs that Cloudera Manager used to access HDFS and Impala were stored in the /tmp directory. This left them exposed to deletion either manually or automatically. Somehow, it appears that the keytab file is missing in your case and the way to regenerate that keytab file is to restart Cloudera Manager.
In Cloudera Manager 5.5, the keytab is stored in /var/run/cloudera-scm-server