Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

Solved Go to solution

aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

New Contributor

Failed to start Firehose
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:274)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:309)
at com.cloudera.enterprise.AbstractCDHVersionAwarePeriodicService.<init>(AbstractCDHVersionAwarePeriodicService.java:73)
at com.cloudera.cmon.firehose.JobTrackerPoller.<init>(JobTrackerPoller.java:192)
at com.cloudera.cmon.firehose.TreeJobTrackerPoller.<init>(TreeJobTrackerPoller.java:45)
at com.cloudera.cmon.firehose.FirehosePipeline.createSecurityAwarePollers(FirehosePipeline.java:333)
at com.cloudera.cmon.firehose.FirehosePipeline.<init>(FirehosePipeline.java:215)
at com.cloudera.cmon.firehose.FirehosePipeline.<init>(FirehosePipeline.java:385)
at com.cloudera.cmon.firehose.Firehose.<init>(Firehose.java:241)
at com.cloudera.cmon.firehose.Main.main(Main.java:510)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:268)
... 9 more
Caused by: java.lang.RuntimeException: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:491)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:855)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:279)
at com.cloudera.cmf.cdh4client.CDH4ObjectFactoryImpl.login(CDH4ObjectFactoryImpl.java:188)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:485)
... 5 more
Caused by: javax.security.auth.login.LoginException: Connection refused
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:767)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:846)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.krb5.internal.TCPClient.<init>(NetClient.java:65)
at sun.security.krb5.internal.NetClient.getInstance(NetClient.java:43)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:372)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:343)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.krb5.KdcComm.send(KdcComm.java:327)
at sun.security.krb5.KdcComm.send(KdcComm.java:219)
at sun.security.krb5.KdcComm.send(KdcComm.java:191)
at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:319)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:364)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:735)
... 21 more

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

New Contributor

thanks a lot,but not the answer

when i set HADOOP_OPTS="-Dsun.security.krb5.debug=true" ,i found it connect kdc using tcp, then i check the kdc config,kdc only listined on udp, so "conneciton confused"

3 REPLIES 3

Re: aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

Super Collaborator

make sure forward and reverse lookup are working correctly on all nodes (per discussion here)

 

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

 

Make sure you deployed client configurations through CM after enabling kerberos and restarting services too (probably not the issue, but important to remember)

 

If management services were not down when you enabled kerberos, it might be necessary to kill them, try to stop them and all cluster services first, and then check if they came down clean with a ps -ef | grep java | grep cloudera

Highlighted

Re: aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

New Contributor

thanks a lot,but not the answer

when i set HADOOP_OPTS="-Dsun.security.krb5.debug=true" ,i found it connect kdc using tcp, then i check the kdc config,kdc only listined on udp, so "conneciton confused"

Re: aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

New Contributor