Reply
New Contributor
Posts: 4
Registered: ‎11-22-2015

HDFS deployment problems

[ Edited ]

I'm trying to use virtualbox and Cloudera Manager 5.5 to set up a new cluster with four nodes "hadoop[1-4]". Hadoop HDFS deployment just does not work. I checked /dfs/dn permission on every data node. It is

 

"drwxr-xr-x  2 cloudera-scm cloudera-scm 4096 Nov 22 18:36 dn"

 

I have spent hours debugging this. Please help!

I used single user setup and the error msg is the following:

 

2015-11-22 18:20:25,526 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/dfs/dn"
    at org.apache.hadoop.hdfs.server.datanode.DataNode.getDataDirsFromURIs(DataNode.java:1821)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1791)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1728)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1751)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1904)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1925)

 

 

2015-11-22 18:25:02,310 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: failed to stat a path component: '/var/run/hdfs-sockets'.  error code 2 (No such file or directory)
    at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)
    at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:191)
    at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:42)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:603)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:570)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:741)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:344)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1795)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1728)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1751)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1904)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1925)

New Contributor
Posts: 2
Registered: ‎01-13-2016

Re: HDFS deployment problems

Yes same happened to me, also even if you fix /var/run/hdfs-sockets it will removed on reboot

New Contributor
Posts: 2
Registered: ‎01-13-2016

Re: HDFS deployment problems

changed UNIX Domain Socket path value in cluster configuration Configuration - Local Data Directories and Files to be a different directory than /var/run/hdfs-sockets so it would not be deleted on reboot

The save triggered a push of the updated config to all nodes

Announcements