Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hadoop/Kerberos issue

avatar
Rising Star

I have a two node cluster – apache hadoop.

 

On my namenode host, datanode is also running.

 

I enabled kerberos security on the cluster using : http://queryio.com/hadoop-big-data-docs/hadoop-big-data-admin-guide/queryio/hadoop-security-setup-ke...

 

Host names are Hadoop2.hitex.com and Hadoop4.hitex.com. Due to initial caps in hostname, I saw issues in the log where it was using all lower case for host name – so I hard coded the host names in hdfs-site.xml

 

My user:group is hitex:hitex

 

The namenode and secondary namenode startup fine. But when I try to start the datanode (using sudo) on same host, it gives error as below. I think datanode is not able to connect to namenode :
2015-11-24 18:06:45,641 INFO org.mortbay.log: jetty-6.1.26
2015-11-24 18:06:46,870 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:1006
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = hitex
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = hitex
2015-11-24 18:06:47,111 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-11-24 18:06:47,188 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2015-11-24 18:06:47,494 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2015-11-24 18:06:47,844 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2015-11-24 18:06:48,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2015-11-24 18:06:48,102 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hitex/Desktop/Hadoop/nodedata/datanode_dir should be specified as a URI in configuration files. Please update hdfs configuration.
2015-11-24 18:06:48,118 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to Hadoop2.hitex.com/192.168.11.21:9000 starting to offer service
2015-11-24 18:06:48,178 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-11-24 18:06:48,202 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2015-11-24 18:06:48,928 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:48,937 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:53,954 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:53,956 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:58,970 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:58,971 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000

 

In above log I can see that it is using lower case host name (datanode trying to connect to namenode) – but I rechecked everywhere and cannot find where it is defined so. As noted earlier I have hard coded hostname in hdfs-site.xml

 

Appreciate any help in resolving this – I have already spent 2 days on this without any breakthrough. Thanks.

1 ACCEPTED SOLUTION

avatar
Hi,

Unfortunately, there's a known bug in hadoop where any capital letters in host names makes Kerberos not work. The only thing you can do is change your host names to be all lowercase.

Thanks,
Darren

View solution in original post

4 REPLIES 4

avatar
Rising Star

I would appreciate any insights on how to handle the initial caps in hostname in kerberos setup - like the 'H' in Hadoop2.hitex.com which gets modified to hadoop2.hitex.com and so fails.

avatar
Hi,

Unfortunately, there's a known bug in hadoop where any capital letters in host names makes Kerberos not work. The only thing you can do is change your host names to be all lowercase.

Thanks,
Darren

avatar
Rising Star
Thanks for the clarification Darren.

avatar
New Contributor
I agree with you