Reply
Explorer
Posts: 35
Registered: ‎11-24-2015
Accepted Solution

Hadoop/Kerberos issue

I have a two node cluster – apache hadoop.

 

On my namenode host, datanode is also running.

 

I enabled kerberos security on the cluster using : http://queryio.com/hadoop-big-data-docs/hadoop-big-data-admin-guide/queryio/hadoop-security-setup-ke...

 

Host names are Hadoop2.hitex.com and Hadoop4.hitex.com. Due to initial caps in hostname, I saw issues in the log where it was using all lower case for host name – so I hard coded the host names in hdfs-site.xml

 

My user:group is hitex:hitex

 

The namenode and secondary namenode startup fine. But when I try to start the datanode (using sudo) on same host, it gives error as below. I think datanode is not able to connect to namenode :
2015-11-24 18:06:45,641 INFO org.mortbay.log: jetty-6.1.26
2015-11-24 18:06:46,870 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:1006
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = hitex
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = hitex
2015-11-24 18:06:47,111 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-11-24 18:06:47,188 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2015-11-24 18:06:47,494 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2015-11-24 18:06:47,844 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2015-11-24 18:06:48,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2015-11-24 18:06:48,102 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hitex/Desktop/Hadoop/nodedata/datanode_dir should be specified as a URI in configuration files. Please update hdfs configuration.
2015-11-24 18:06:48,118 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to Hadoop2.hitex.com/192.168.11.21:9000 starting to offer service
2015-11-24 18:06:48,178 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-11-24 18:06:48,202 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2015-11-24 18:06:48,928 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:48,937 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:53,954 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:53,956 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:58,970 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:58,971 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000

 

In above log I can see that it is using lower case host name (datanode trying to connect to namenode) – but I rechecked everywhere and cannot find where it is defined so. As noted earlier I have hard coded hostname in hdfs-site.xml

 

Appreciate any help in resolving this – I have already spent 2 days on this without any breakthrough. Thanks.

Explorer
Posts: 35
Registered: ‎11-24-2015

Re: Hadoop/Kerberos issue

I would appreciate any insights on how to handle the initial caps in hostname in kerberos setup - like the 'H' in Hadoop2.hitex.com which gets modified to hadoop2.hitex.com and so fails.

Cloudera Employee
Posts: 508
Registered: ‎07-30-2013

Re: Hadoop/Kerberos issue

Hi,

Unfortunately, there's a known bug in hadoop where any capital letters in host names makes Kerberos not work. The only thing you can do is change your host names to be all lowercase.

Thanks,
Darren
Explorer
Posts: 35
Registered: ‎11-24-2015

Re: Hadoop/Kerberos issue

Thanks for the clarification Darren.
New Contributor
Posts: 1
Registered: ‎01-18-2019

Re: Hadoop/Kerberos issue

I agree with you
Announcements