Created 07-24-2018 04:29 PM
Hi,
I am getting the below error after checking log ( log while starting datanode). Datanode stoping immediately after start.
tailf /var/log/hadoop/hdfs/hadoop-hdfs-datanode-[hostname].log
2018-07-24 10:45:18,282 INFO common.Storage (Storage.java:tryLock(776)) - Lock on /mnt/dn/sdl/datanode/in_use.lock acquired by nodename 55141@dat01.node 2018-07-24 10:45:18,283 WARN common.Storage (DataStorage.java:loadDataStorage(449)) - Failed to add storage directory [DISK]file:/mnt/dn/sdl/datanode/ java.io.FileNotFoundException: /mnt/dn/sdl/datanode/current/VERSION (Permission denied) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) at org.apache.hadoop.hdfs.server.common.StorageInfo.readPropertiesFile(StorageInfo.java:245) at org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:231) at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:779) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:322) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:438) at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:417) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:595) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1543) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1504) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:319) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:272) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:768) at java.lang.Thread.run(Thread.java:748) 2018-07-24 10:45:18,283 ERROR datanode.DataNode (BPServiceActor.java:run(780)) - Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to nam02.node/192.168.19.3:8020. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:596) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1543) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1504) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:319) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:272) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:768) at java.lang.Thread.run(Thread.java:748)
Created 07-24-2018 05:58 PM
@Rambabu Chamakuri It seems that the permissions on the VERSION file are wrong, see below:
java.io.FileNotFoundException: /mnt/dn/sdl/datanode/current/VERSION (Permission denied)
Check the permission on this VERSION file:
ls -lh /mnt/dn/sdl/datanode/current/VERSION
The file should be owned by "hdfs:hdfs" and permissions set to 644. If they are not then change accordingly:
chown hdfs:hdfs /mnt/dn/sdl/datanode/current/VERSION chmod 644 /mnt/dn/sdl/datanode/current/VERSION
And restart the Datanode.
Let me know if it help in solving your issue
Created 07-25-2018 05:00 AM
thanks for your reply. I checked the permissions it was fine (The file owned by "hdfs:hdfs" and permissions set to 644).
The node was out of service for an extended time, So I followed the below steps
Now Datanode is up and live.... Thanks for hortonworks help and contribution.
Reference : https://community.hortonworks.com/questions/192751/databode-uuid-unassigned.html