question Re: Single Datanode start fails ? in Archives of Support Questions (Read Only)

Single Datanode start fails ?

btandel — Mon, 15 May 2017 18:03:56 GMT

Guys we have kerberized cluster with hdp 2.6. We had to stop and restart the machines over the weekend. But while restarting we are facing issue with just one datanode, the error is

2017-05-15 09:42:41,555 ERROR datanode.DataNode (DataNode.java:secureMain(2691)) - Exception in secureMain java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP. Using privileged resources in combination with SASL RPC data transfer protection is not supported. at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1354) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1224) at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:456) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2590) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2492) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2539) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2684) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2708) 2017-05-15 09:42:41,557 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2017-05-15 09:42:41,560 INFO datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:

We dont have secureDatanode on.

Can anyone help me out here please ?

our dfs.datanode.address is 0.0.0.0:1019

Re: Single Datanode start fails ?

snukavarapu — Mon, 15 May 2017 21:52:35 GMT

@btandel -

Possible cause for this error is kerberos file misconfiguration on the datanode.

Verify if the JCE security libraries are deployed correctly on the datanode and krb5.conf file is set the same way as other datanodes.

Re: Single Datanode start fails ?

btandel — Tue, 16 May 2017 14:20:29 GMT

@snukavarapu Thanks for quick response.

JCE security libraries are deployed same as other nodes and also krb5.conf is taken care by Vasd tool.

Re: Single Datanode start fails ?

namaheshwari — Thu, 18 May 2017 03:26:57 GMT

Can you verify if the property values of both the properties is less than 1023 i.e. priviliged port

dfs.datanode.address
dfs.datanode.http.address

Re: Single Datanode start fails ?

Shelton — Thu, 18 May 2017 12:35:43 GMT

@btandel

Have a look at this

Re: Single Datanode start fails ?

btandel — Mon, 22 May 2017 00:45:29 GMT

Thanks a lot @Geoffrey Shelton Okot, @Namit Maheshwari and @snukavarapu for the support. However we found the issue, one of the internal port was already in used and so it was not coming up. We finally resolved by killing the process and restarting the datanode.