I have a two node cluster – apache hadoop.
On my namenode host, datanode is also running.
I enabled kerberos security on the cluster using : http://queryio.com/hadoop-big-data-docs/hadoop-big-data-admin-guide/queryio/hadoop-security-setup-ke...
Host names are Hadoop2.hitex.com and Hadoop4.hitex.com. Due to initial caps in hostname, I saw issues in the log where it was using all lower case for host name – so I hard coded the host names in hdfs-site.xml
My user:group is hitex:hitex
The namenode and secondary namenode startup fine. But when I try to start the datanode (using sudo) on same host, it gives error as below. I think datanode is not able to connect to namenode :
2015-11-24 18:06:45,641 INFO org.mortbay.log: jetty-6.1.26
2015-11-24 18:06:46,870 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:1006
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = hitex
2015-11-24 18:06:46,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = hitex
2015-11-24 18:06:47,111 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-11-24 18:06:47,188 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2015-11-24 18:06:47,494 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2015-11-24 18:06:47,844 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2015-11-24 18:06:48,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2015-11-24 18:06:48,102 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hitex/Desktop/Hadoop/nodedata/datanode_dir should be specified as a URI in configuration files. Please update hdfs configuration.
2015-11-24 18:06:48,118 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to Hadoop2.hitex.com/192.168.11.21:9000 starting to offer service
2015-11-24 18:06:48,178 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-11-24 18:06:48,202 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2015-11-24 18:06:48,928 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:48,937 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:53,954 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:53,956 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
2015-11-24 18:06:58,970 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hitex/hadoop2.hitex.com@hitex.com
2015-11-24 18:06:58,971 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Hadoop2.hitex.com/192.168.11.21:9000
In above log I can see that it is using lower case host name (datanode trying to connect to namenode) – but I rechecked everywhere and cannot find where it is defined so. As noted earlier I have hard coded hostname in hdfs-site.xml
Appreciate any help in resolving this – I have already spent 2 days on this without any breakthrough. Thanks.
