Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Issue with mixed case server names

avatar
Explorer

Having this issue in CM4.7, just wondering if its fixed in future release?

 

because my servers are mixed case, it failed to find the proper principal in the db:

 

Exception in secureMain

java.io.IOException: Login failure for hdfs/1620-megatron.bi2r.leidos.com@BI2R.LEIDOS.COM from keytab hdfs.keytab

                at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:825)

                at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:279)

                at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:243)

                at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1726)

                at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1751)

                at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1904)

                at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:135)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:188)

Caused by: javax.security.auth.login.LoginException: Checksum failed

                at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:763)

                at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at javax.security.auth.login.LoginContext.invoke(LoginContext.java:784)

                at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)

                at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721)

                at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719)

                at java.security.AccessController.doPrivileged(Native Method)

                at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718)

                at javax.security.auth.login.LoginContext.login(LoginContext.java:590)

                at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:816)

                ... 11 more

 

As you can see here, regenerating the principals created new keytabs, for the principal with mixed case hostnames:

 

[root@1620-authentication krb5kdc]# kadmin.local

Authenticating as principal root/admin@BI2R.LEIDOS.COM with password.

kadmin.local:  getprincs

HTTP/1620-Megatron.bi2r.leidos.com@BI2R.LEIDOS.COM

HTTP/1620-Node6.bi2r.leidos.com@BI2R.LEIDOS.COM

HTTP/1620-Scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM

HTTP/1620-cdh4u5.bi2r.leidos.com@BI2R.LEIDOS.COM

K/M@BI2R.LEIDOS.COM

bir/admin@BI2R.LEIDOS.COM

cloudera-scm/admin@BI2R.LEIDOS.COM

hdfs/1620-Megatron.bi2r.leidos.com@BI2R.LEIDOS.COM

hdfs/1620-Node6.bi2r.leidos.com@BI2R.LEIDOS.COM

hdfs/1620-Scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM

hdfs/1620-cdh4u5.bi2r.leidos.com@BI2R.LEIDOS.COM

hdfs/1620-megatron.bi2r.leidos.com@BI2R.LEIDOS.COM

hdfs/1620-node6.bi2r.leidos.com@BI2R.LEIDOS.COM

hdfs/1620-scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM

 ...

 

The fix was to delete the principals I renamed last time, and then rename them:

 

kadmin.local:  delprinc hdfs/1620-megatron.bi2r.leidos.com

Are you sure you want to delete the principal "hdfs/1620-megatron.bi2r.leidos.com@BI2R.LEIDOS.COM"? (yes/no): yes

Principal "hdfs/1620-megatron.bi2r.leidos.com@BI2R.LEIDOS.COM" deleted.

Make sure that you have removed this principal from all ACLs before reusing.

kadmin.local:  delprinc hdfs/1620-node6.bi2r.leidos.com

Are you sure you want to delete the principal "hdfs/1620-node6.bi2r.leidos.com@BI2R.LEIDOS.COM"? (yes/no): yes

Principal "hdfs/1620-node6.bi2r.leidos.com@BI2R.LEIDOS.COM" deleted.

Make sure that you have removed this principal from all ACLs before reusing.

kadmin.local:  delprinc hdfs/1620-scavenger.bi2r.leidos.com

Are you sure you want to delete the principal "hdfs/1620-scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM"? (yes/no): yes

Principal "hdfs/1620-scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM" deleted.

Make sure that you have removed this principal from all ACLs before reusing.

 

kadmin.local:  renprinc hdfs/1620-Megatron.bi2r.leidos.com hdfs/1620-megatron.bi2r.leidos.com

Are you sure you want to rename the principal "hdfs/1620-Megatron.bi2r.leidos.com@BI2R.LEIDOS.COM" to "hdfs/1620-megatron.bi2r.leidos.com@BI2R.LEIDOS.COM"? (yes/no): yes

Principal "hdfs/1620-Megatron.bi2r.leidos.com@BI2R.LEIDOS.COM" renamed to "hdfs/1620-megatron.bi2r.leidos.com@BI2R.LEIDOS.COM".

Make sure that you have removed the old principal from all ACLs before reusing.

kadmin.local:  renprinc hdfs/1620-Node6.bi2r.leidos.com hdfs/1620-node6.bi2r.leidos.com

Are you sure you want to rename the principal "hdfs/1620-Node6.bi2r.leidos.com@BI2R.LEIDOS.COM" to "hdfs/1620-node6.bi2r.leidos.com@BI2R.LEIDOS.COM"? (yes/no): yes

Principal "hdfs/1620-Node6.bi2r.leidos.com@BI2R.LEIDOS.COM" renamed to "hdfs/1620-node6.bi2r.leidos.com@BI2R.LEIDOS.COM".

Make sure that you have removed the old principal from all ACLs before reusing.

kadmin.local:  renprinc hdfs/1620-Scavenger.bi2r.leidos.com hdfs/1620-scavenger.bi2r.leidos.com

Are you sure you want to rename the principal "hdfs/1620-Scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM" to "hdfs/1620-scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM"? (yes/no): yes

Principal "hdfs/1620-Scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM" renamed to "hdfs/1620-scavenger.bi2r.leidos.com@BI2R.LEIDOS.COM".

Make sure that you have removed the old principal from all ACLs before reusing.

 

 

Interestingly enough, I only have this issue with hdfs and mapred roles… zookeeper, oozie and HTTP all respect the mixed case.  

http://championofcyrodiil.blogspot.com
1 ACCEPTED SOLUTION

avatar

Unfortunately, host names with capital letters will always hit this problem. CM respects the original host name capitalization, but hadoop converts it all to lowercase.

 

You'll have to pick lowercase host names. You might also be able to change your agent.ini on each host to override the hostname to be the lowercase name, haven't tried that though.

 

Thanks,

Darren

View solution in original post

2 REPLIES 2

avatar
Explorer

I guess its and issue with jdk, since the hostnames are fetched that way.

 

http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/h...

 

 

http://championofcyrodiil.blogspot.com

avatar

Unfortunately, host names with capital letters will always hit this problem. CM respects the original host name capitalization, but hadoop converts it all to lowercase.

 

You'll have to pick lowercase host names. You might also be able to change your agent.ini on each host to override the hostname to be the lowercase name, haven't tried that though.

 

Thanks,

Darren