Support Questions

Find answers, ask questions, and share your expertise

Single Datanode start fails ?

avatar
Explorer

Guys we have kerberized cluster with hdp 2.6. We had to stop and restart the machines over the weekend. But while restarting we are facing issue with just one datanode, the error is

2017-05-15 09:42:41,555 ERROR datanode.DataNode (DataNode.java:secureMain(2691)) - Exception in secureMain java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP.  Using privileged resources in combination with SASL RPC data transfer protection is not supported.   at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1354)   at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1224)   at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:456)   at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2590)   at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2492)   at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2539)   at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2684)   at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2708) 2017-05-15 09:42:41,557 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2017-05-15 09:42:41,560 INFO  datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:

We dont have secureDatanode on.

Can anyone help me out here please ?

our dfs.datanode.address is 0.0.0.0:1019

1 ACCEPTED SOLUTION

avatar
Explorer

Thanks a lot @Geoffrey Shelton Okot, @Namit Maheshwari and @snukavarapu for the support. However we found the issue, one of the internal port was already in used and so it was not coming up. We finally resolved by killing the process and restarting the datanode.

View solution in original post

5 REPLIES 5

avatar
Contributor

@btandel -

Possible cause for this error is kerberos file misconfiguration on the datanode.

Verify if the JCE security libraries are deployed correctly on the datanode and krb5.conf file is set the same way as other datanodes.

avatar
Explorer

@snukavarapu Thanks for quick response.

JCE security libraries are deployed same as other nodes and also krb5.conf is taken care by Vasd tool.

avatar

Can you verify if the property values of both the properties is less than 1023 i.e. priviliged port

dfs.datanode.address
dfs.datanode.http.address

avatar
Master Mentor

@btandel

Have a look at this

avatar
Explorer

Thanks a lot @Geoffrey Shelton Okot, @Namit Maheshwari and @snukavarapu for the support. However we found the issue, one of the internal port was already in used and so it was not coming up. We finally resolved by killing the process and restarting the datanode.