Single Datanode start fails ?

[#1]

Guys we have kerberized cluster with hdp 2.6. We had to stop and restart the machines over the weekend. But while restarting we are facing issue with just one datanode, the error is

2017-05-15 09:42:41,555 ERROR datanode.DataNode (DataNode.java:secureMain(2691)) - Exception in secureMain java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP.  Using privileged resources in combination with SASL RPC data transfer protection is not supported.   at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1354)   at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1224)   at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:456)   at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2590)   at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2492)   at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2539)   at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2684)   at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2708) 2017-05-15 09:42:41,557 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2017-05-15 09:42:41,560 INFO  datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:

We dont have secureDatanode on.

Can anyone help me out here please ?

our dfs.datanode.address is 0.0.0.0:1019

[#2]

@btandel -

Possible cause for this error is kerberos file misconfiguration on the datanode.

Verify if the JCE security libraries are deployed correctly on the datanode and krb5.conf file is set the same way as other datanodes.

[#3]

@snukavarapu Thanks for quick response.

JCE security libraries are deployed same as other nodes and also krb5.conf is taken care by Vasd tool.

[#4]

Can you verify if the property values of both the properties is less than 1023 i.e. priviliged port

dfs.datanode.address
dfs.datanode.http.address

[#5]

@btandel

Have a look at this

[#6 · Accepted solution]

Thanks a lot @Geoffrey Shelton Okot, @Namit Maheshwari and @snukavarapu for the support. However we found the issue, one of the internal port was already in used and so it was not coming up. We finally resolved by killing the process and restarting the datanode.