Reply
New Contributor
Posts: 4
Registered: ‎09-10-2014
Accepted Solution

aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

Failed to start Firehose
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:274)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:309)
at com.cloudera.enterprise.AbstractCDHVersionAwarePeriodicService.<init>(AbstractCDHVersionAwarePeriodicService.java:73)
at com.cloudera.cmon.firehose.JobTrackerPoller.<init>(JobTrackerPoller.java:192)
at com.cloudera.cmon.firehose.TreeJobTrackerPoller.<init>(TreeJobTrackerPoller.java:45)
at com.cloudera.cmon.firehose.FirehosePipeline.createSecurityAwarePollers(FirehosePipeline.java:333)
at com.cloudera.cmon.firehose.FirehosePipeline.<init>(FirehosePipeline.java:215)
at com.cloudera.cmon.firehose.FirehosePipeline.<init>(FirehosePipeline.java:385)
at com.cloudera.cmon.firehose.Firehose.<init>(Firehose.java:241)
at com.cloudera.cmon.firehose.Main.main(Main.java:510)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:268)
... 9 more
Caused by: java.lang.RuntimeException: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:491)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Login failure for hue/hadoop-pcrm-06.abc@ABC from keytab hue.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:855)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:279)
at com.cloudera.cmf.cdh4client.CDH4ObjectFactoryImpl.login(CDH4ObjectFactoryImpl.java:188)
at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:485)
... 5 more
Caused by: javax.security.auth.login.LoginException: Connection refused
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:767)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:846)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.krb5.internal.TCPClient.<init>(NetClient.java:65)
at sun.security.krb5.internal.NetClient.getInstance(NetClient.java:43)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:372)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:343)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.krb5.KdcComm.send(KdcComm.java:327)
at sun.security.krb5.KdcComm.send(KdcComm.java:219)
at sun.security.krb5.KdcComm.send(KdcComm.java:191)
at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:319)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:364)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:735)
... 21 more

Cloudera Employee
Posts: 229
Registered: ‎09-23-2013

Re: aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

make sure forward and reverse lookup are working correctly on all nodes (per discussion here)

 

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

 

Make sure you deployed client configurations through CM after enabling kerberos and restarting services too (probably not the issue, but important to remember)

 

If management services were not down when you enabled kerberos, it might be necessary to kill them, try to stop them and all cluster services first, and then check if they came down clean with a ps -ef | grep java | grep cloudera

New Contributor
Posts: 4
Registered: ‎09-10-2014

Re: aftper config kerberos on CDH5,Service Monitor,HDFS can not start.

thanks a lot,but not the answer

when i set HADOOP_OPTS="-Dsun.security.krb5.debug=true" ,i found it connect kdc using tcp, then i check the kdc config,kdc only listined on udp, so "conneciton confused"

Highlighted
New Contributor
Posts: 1
Registered: ‎08-26-2016

Re: aftper config kerberos on CDH5,Service Monitor,HDFS can not start.