Can't start Activity Monitor and Zookeeper after enabling Kerberos - error "no password provided"

[#1]

I've enabled Kerberos using AD as KDC, and I can kinit with both the zookeeper and cm keytabs mentioned on the errors below.

What could be wrong?

Some things I tried/noticed:

- regenerated the keytabs, including removing the principals from AD.

- guaranteed the permissions on the keytab files are correct: cloudera-scm:cloudera-scm for cm.keytab and zookeeper:zookeeper on zookeeper.keytab

- the jaas.conf file has the keytab entry with the filename without the full path, but I think this is not an issue as the directory is auto-generated at each start.

Successful kinit with zookeeper.keytab:

Error on Zookeeper:

Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided
	at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

Successful kinit with cmon.keytab:

Error on starting Activity Monitor:

Failed to start Firehose
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Login failure for hue/cldra-pr-edge2.xxx@XXX from keytab cmon.keytab
	at com.google.common.base.Throwables.propagate(Throwables.java:160)
	at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:293)
	at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:349)
	at com.cloudera.enterprise.AbstractCDHVersionAwarePeriodicService.<init>(AbstractCDHVersionAwarePeriodicService.java:73)
	at com.cloudera.cmon.firehose.JobTrackerPoller.<init>(JobTrackerPoller.java:192)
	at com.cloudera.cmon.firehose.TreeJobTrackerPoller.<init>(TreeJobTrackerPoller.java:45)
	at com.cloudera.cmon.firehose.FirehosePipeline.createSecurityAwarePollers(FirehosePipeline.java:332)
	at com.cloudera.cmon.firehose.FirehosePipeline.<init>(FirehosePipeline.java:214)
	at com.cloudera.cmon.firehose.FirehosePipeline.<init>(FirehosePipeline.java:384)
	at com.cloudera.cmon.firehose.Firehose.<init>(Firehose.java:262)
	at com.cloudera.cmon.firehose.Main.main(Main.java:541)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Login failure for hue/cldra-pr-edge2.xxx@XXX from keytab cmon.keytab
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:287)
	... 9 more
Caused by: java.lang.RuntimeException: java.io.IOException: Login failure for hue/cldra-pr-edge2xxx@XXXX from keytab cmon.keytab
	at com.google.common.base.Throwables.propagate(Throwables.java:160)
	at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:584)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Login failure for hue/cldra-pr-edge2.xxx@XXX from keytab cmon.keytab
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:855)
	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:279)
	at com.cloudera.cmf.cdh4client.CDH4ObjectFactoryImpl.login(CDH4ObjectFactoryImpl.java:190)
	at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:578)
	... 5 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

	at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:846)
	... 8 more

[#2 · Accepted solution]

@dougspadottoemc,

Thank you for supplying all that information.

A couple possible causes I can think of:

(1)

You are using AES256 encryption for kerberos but the JDK you are using is not configured to support AES256.

- See if you have encryption types configured in your /etc/krb5.conf

- use "klist -kte zookeeper.keytab" to view the encryption types listed in the keytab

To check this, try this in Cloudera Manager:

- Navigate to Administration --> Security

- Click Security Inspector button

- When the check completes, check the results and make sure you see:

All hosts have Java configured with unlimited-strength encryption

If you do not see that and you have limited encryption, that can explain why "kinit" works but Java cann't read the keytab properly.  MIT does not have AES256 restrictions.

(2)

Since you redacted your domain and realm information, I can't tell for sure, but I do recall this type of issue happening in the past when Cloudera Manager had a different REALM configured than the one in the keytabs.  I don't think this is it for you, but thought I'd mention it just in case.

[#3]

Thanks @bgooley!

The inspector showed the unlimited-strength check as ok, but I was suspicious of the same thing and asked the admin to make sure the Unlimited JCE policy jar files were copied.

After he copied them, Zookeeper was started successfuly!

I'll mark your answer as the solution, even though I arrived at the same conclusion just hours before your reply! 🙂

[#4]

Thanks, @dougspadottoemc!  I am just glad you tracked down the problem and can go back to having fun with hadoop 😉

[#5]

This definitely was the issue and the fix for me. I saw both jar files already in the java/lib/security dir and failed to replace them with the downloaded UnlimitedJCEPolicyJDK8 jar files.

It took me a long time to get back to replacing the files. After doing so, I restarted CMS successfuly.

Thanks for the post!