Hadoop/Kerberos issue

vtpcnk — Fri, 16 Sep 2022 09:50:13 GMT

I have a two node cluster – apache hadoop.

On my namenode host, datanode is also running.

I enabled kerberos security on the cluster using : http://queryio.com/hadoop-big-data-docs/hadoop-big-data-admin-guide/queryio/hadoop-security-setup-kerberos.html

Host names are Hadoop2.hitex.com and Hadoop4.hitex.com. Due to initial caps in hostname, I saw issues in the log where it was using all lower case for host name – so I hard coded the host names in hdfs-site.xml

My user:group is hitex:hitex

The namenode and secondary namenode startup fine. But when I try to start the datanode (using sudo) on same host, it gives error as below. I think datanode is not able to connect to namenode :
2015-11-24 18:06:45,641 INFO org.mortbay.log: jetty-6.1.26
2015-11-24 18:06:46,870 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:1006
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = hitex
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = hitex
2015-11-24 18:06:47,111 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-11-24 18:06:47,188 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2015-11-24 18:06:47,494 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2015-11-24 18:06:47,844 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2015-11-24 18:06:48,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2015-11-24 18:06:48,102 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hitex/Desktop/Hadoop/nodedata/datanode_dir should be specified as a URI in configuration files. Please update hdfs configuration.
2015-11-24 18:06:48,118 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to Hadoop2.hitex.com/192.168.11.21:9000 starting to offer service
2015-11-24 18:06:48,178 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-11-24 18:06:48,202 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2015-11-24 18:06:48,928 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:48,937 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:53,954 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:53,956 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:58,970 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:58,971 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000

In above log I can see that it is using lower case host name (datanode trying to connect to namenode) – but I rechecked everywhere and cannot find where it is defined so. As noted earlier I have hard coded hostname in hdfs-site.xml

Appreciate any help in resolving this – I have already spent 2 days on this without any breakthrough. Thanks.

Re: Hadoop/Kerberos issue

vtpcnk — Tue, 24 Nov 2015 16:00:00 GMT

I would appreciate any insights on how to handle the initial caps in hostname in kerberos setup - like the 'H' in Hadoop2.hitex.com which gets modified to hadoop2.hitex.com and so fails.

Re: Hadoop/Kerberos issue

Darren — Tue, 24 Nov 2015 16:40:35 GMT

Hi,

Unfortunately, there's a known bug in hadoop where any capital letters in host names makes Kerberos not work. The only thing you can do is change your host names to be all lowercase.

Thanks,
Darren

Re: Hadoop/Kerberos issue

vtpcnk — Tue, 24 Nov 2015 21:03:52 GMT

Thanks for the clarification Darren.

Re: Hadoop/Kerberos issue

kaduswapnali456 — Sat, 19 Jan 2019 05:08:14 GMT

I agree with you

question Re: Hadoop/Kerberos issue in Archives of Support Questions (Read Only)

Hadoop/Kerberos issue

Re: Hadoop/Kerberos issue

Re: Hadoop/Kerberos issue

Re: Hadoop/Kerberos issue

Re: Hadoop/Kerberos issue