Datanodes not starting after disabling Kerberos

[#1]

Hi,

I disabled Kerberos (setting Service HDFS => Configuration => Authentication type "simple") while all service are stopped. Afterwards I wanted to start service HDFS, but the Datanodes fail with the error:

Exception in secureMain
java.io.IOException: Failed on local exception: java.net.SocketException: Permission denied; Host Details : local host is: "hadoop-pg-4.cluster"; destination host is: (unknown):0;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
at org.apache.hadoop.ipc.Server.bind(Server.java:403)
at org.apache.hadoop.ipc.Server.bind(Server.java:375)
at org.apache.hadoop.hdfs.net.TcpPeerServer.<init>(TcpPeerServer.java:106)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:555)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:741)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:344)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1795)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1728)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1751)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1904)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1925)
Caused by: java.net.SocketException: Permission denied
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:386)
... 10 more

on the datanodes theres is no hadoop related process running and nothing is listening on the required ports.

How do I successfully disable Kerberos and start my cluster afterwards?

CDH4.6 / CM4.8

regards, Gerd

[#2 · Accepted solution]

Hi Gerd,

Are your data nodes configured to use ports < 1024? You'll need to rever these to their default ports.

Here's the doc page from the enable security guide that talks about ports. You'll need to do the opposite.

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Configuring-Hadoop-Secu...

Thanks,

Darren

[#3]

Hi Darren,

GOTCHA 😉

reverting both ports back to default solved the problem, many thanks !

[#4]

Hi Darren,

I'm facing the same problem. Eventhough datanode port points to 1004. other than port anything has to changed?

i'm using cdh5.5.2. 

Error,

Exception in secureMain java.io.IOException: Failed on local exception: java.net.SocketException: Permission denied; Host Details : local host is: "hadoop03.rlpl.com"; destination host is: (unknown):0; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Server.bind(Server.java:424) at org.apache.hadoop.ipc.Server.bind(Server.java:396) at org.apache.hadoop.hdfs.net.TcpPeerServer.<init>(TcpPeerServer.java:111) at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:921) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1144) at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:453) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2402) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2289) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2336) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2513) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2537) Caused by: java.net.SocketException: Permission denied at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.hadoop.ipc.Server.bind(Server.java:407)

Thanks,

   Venkat

[#5]

Hi, Venkat,

maybe this will help you.

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos/td-p/19654