Datanode doesn't start. Error: The path component: '/' in '/var/lib/hadoop-hdfs/dn_socket' has permissions 0777

[#1]

After a successful fresh Installation of HDP 3.1.0 on a 2 node Ubuntu 18.0.4 instances, we were able to get all the services up and running.

But after a night off, Datanode doesn't start. Following is the error:

ERROR datanode.DataNode (DataNode.java:secureMain(2883)) - Exception in secureMain
java.io.IOException: The path component: '/' in '/var/lib/hadoop-hdfs/dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity
    at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)
    at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:193)
    at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1194)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1161)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1416)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:500)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2782)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2690)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2732)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2876)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2900)
2019-04-08 17:36:52,452 INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: java.io.IOException: The path component: '/' in '/var/lib/hadoop-hdfs/dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity
2019-04-08 17:36:52,456 INFO  datanode.DataNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG

I did check the socket file and it's parent directories, I don't see the 777 permission at any level. But still the error appears while starting the datanode.

Couldn't find any solution, so posted here for help. I have also uploaded the complete logs for the datanode.

hadoop-hdfs-datanode.txtRegards,

Vinay MP

[#2 · Accepted solution]

Got the hint logs are exactly pointing the issue.

"ls -ld /" shows me it has 777 permission. Just removed the write permission for group and other users, my issue is solved.

All these while I only checked the permission for the subsequent folders after "/" but the problem lies with "/" itself.