Datanodes failed to start the role because hdfs-sockets cannot be created

[#1]

I was able to complete the installation of CDH-5.5.1.  But, all datanodes have problem starting HDFS service due to the following problem:

2016-02-11 18:18:51,394 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.net.BindException: bind(2) error: Permission denied when trying to bind to '/ssd1/tmp/dfs/socket/dn'
	at org.apache.hadoop.net.unix.DomainSocket.bind0(Native Method)
	at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:191)
	at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:970)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:936)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1144)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:453)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2402)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2289)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2336)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2513)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2537)

The original location was /var/run/hdfs-sockets/dn.  I had the same problem with original location.  So, I changed it to a new location, and I still have the "permission denied" problem.

I installed the Cloudera Manager using the user root. 

[#2]

I am begining to see where the problem lies.  In the Cloudera Manager HDFS config setting, I am using hduser:hadoop as the system user and system group.  After re-starting the HDFS, I saw that /var/run/dfs-socket was created and assigned to hduser:hadoop.

But, I am still getting the error:  

java.io.IOException: the path component: '/var/run/dfs-socket' is owned by a user who is not root and not you.

What I am not clear is when hdfs.sh is run to start the HDFS service, which user is running the script?  Note that root user was used to install the Cloudera Manager and setup all the NameNodes and DataNodes. 

[#3 · Accepted solution]

The problem was solved by changing the system user to root, usergroup to root and shared usergroup to hadoop in the HDFS configuration.  I also removed the original folders named "dfs-socket" in /var/run/ in all DataNodes and let the script create new folders.

[#4]

java.net.BindException: bind(2) error: Permission denied when trying to bind to '/var/run/hdfs-sockets/dn'

Exception in secureMain
java.net.BindException: bind(2) error: Permission denied when trying to bind to '/var/run/hdfs-sockets/dn'
	at org.apache.hadoop.net.unix.DomainSocket.bind0(Native Method)
	at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:191)
	at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:967)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:933)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1137)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:451)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2406)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2293)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2340)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2517)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2541)

Exiting with status 1

SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at icslt.ad.infosys.com/172.25.11.89
************************************************************/

Hi Members,

My problem is not resolving even with ROOT:ROOT permission on 

[root@icslt run]# ls -ltd hdfs-sockets/
drwxr-xr-x 2 root root 4096 May 21 00:52 hdfs-sockets/
[root@icslt run]#

Please suggest its urgent....

Thanks in advance.

Amit Lal

[#5]

[root@Vm91 hadoop-hdfs]# chown hdfs:root -R /var/run/hdfs-sockets

try this one