question Re: datanode + Directory is not writable in Archives of Support Questions (Read Only)

datanode + Directory is not writable

mike_bronson7 — Tue, 24 Apr 2018 00:09:34 GMT

we have ambari cluster HDP version 2.6.0.1

we have issues on worker02 according to the log - hadoop-hdfs-datanode-worker02.sys65.com.log,

2018-04-21 09:02:53,405 WARN  checker.StorageLocationChecker (StorageLocationChecker.java:check(208)) - Exception checking StorageLocation [DISK]file:/grid/sdc/hadoop/hdfs/data/
org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not writable: /grid/sdc/hadoop/hdfs/data

note - from ambari GUI we can see that Data-node on worker02 is down

we can see from the log - Directory is not writable: /grid/sdc/hadoop/hdfs/data the follwing:

STARTUP_MSG: Starting DataNode
STARTUP_MSG:   user = hdfs
STARTUP_MSG:   host = worker02.sys65.com/23.87.23.126
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.7.3.2.6.0.3-8
STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r c6befa0f1e911140cc815e0bab744a6517abddae; compiled by 'jenkins' on 2017-04-01T21:32Z
STARTUP_MSG:   java = 1.8.0_112
************************************************************/
2018-04-21 09:02:52,854 INFO  datanode.DataNode (LogAdapter.java:info(47)) - registered UNIX signal handlers for [TERM, HUP, INT]
2018-04-21 09:02:53,321 INFO  checker.ThrottledAsyncChecker (ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for [DISK]file:/grid/sdb/hadoop/hdfs/data/
2018-04-21 09:02:53,330 INFO  checker.ThrottledAsyncChecker (ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for [DISK]file:/grid/sdc/hadoop/hdfs/data/
2018-04-21 09:02:53,330 INFO  checker.ThrottledAsyncChecker (ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for [DISK]file:/grid/sdd/hadoop/hdfs/data/
2018-04-21 09:02:53,331 INFO  checker.ThrottledAsyncChecker (ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for [DISK]file:/grid/sde/hadoop/hdfs/data/
2018-04-21 09:02:53,331 INFO  checker.ThrottledAsyncChecker (ThrottledAsyncChecker.java:schedule(107)) - Scheduling a check for [DISK]file:/grid/sdf/hadoop/hdfs/data/
2018-04-21 09:02:53,405 WARN  checker.StorageLocationChecker (StorageLocationChecker.java:check(208)) - Exception checking StorageLocation [DISK]file:/grid/sdc/hadoop/hdfs/data/
org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not writable: /grid/sdc/hadoop/hdfs/data
	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:124)
	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:99)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:128)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:44)
	at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:127)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2018-04-21 09:02:53,410 ERROR datanode.DataNode (DataNode.java:secureMain(2691)) - Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 4, volumes configured: 5, volumes failed: 1, volume failures tolerated: 0
	at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:216)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2583)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2492)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2539)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2684)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2708)
2018-04-21 09:02:53,411 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2018-04-21 09:02:53,414 INFO  datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at worker02.sys65.com/23.87.23.126
************************************************************/

<br>

we checked that:

1. all files and folders under - /grid/sdc/hadoop/hdfs/ are with hdfs:hadoop , and that is OK

2. disk - sdc is read and write (rw,noatime,data=ordered) , and that is OK

we suspect that Hard Disk has gone bad , in this case how we check that?

please advice what the other options to resolve this issue ?

Re: datanode + Directory is not writable

Shelton — Tue, 24 Apr 2018 00:58:14 GMT

@Michael Bronson

Could you try umounting and mount that disk? Your disk could have gone bac and the FS is in Read-Only mode Can you also set the failure tolerance to 1

Using Ambari UI--> HDFS-->Configs---Filter in for property "dfs.datanode.failed.volumes.tolerated" set it to 1

Restart stale HDFS services

All should be in order

Re: datanode + Directory is not writable

mike_bronson7 — Tue, 24 Apr 2018 02:39:53 GMT

Dear Geoffrey , we do resboot twice before weeks , but this not help ( when we reboot we do actually remount , about dfs.datanode.failed.volumes.tolerated" set it to 1 , we want to set it to 0 ( we not want loose one disk )

Re: datanode + Directory is not writable

Shelton — Tue, 24 Apr 2018 03:17:40 GMT

@Michael Bronson

There could be a couple of reasons, lete check the obvious have you checked SE Linux on this host? if not

$ echo 0 >/selinux/enforce 
$ cat /selinux/enforce # should output "0"

Read-only filesystem" is not a permissions issue. The mount has become read-only, either because of errors in the filesystem or problems in the device itself. If you run "grep sdc /proc/mounts" you should see it as "ro". There may be some clue as to why in the messages in /var/log/syslog.

Run File system check fsck it will repair some of the errors e.g execute the fsck on an unmounted file system to avoid any data corruption issues. e.g

# fsck /dev/sdc

That should repair the damages.

Re: datanode + Directory is not writable

mike_bronson7 — Tue, 24 Apr 2018 11:34:29 GMT

Dear Geoffrey , /grid/sdc hold HDFS filesystem , dose fsck on that disk not risky? , see also - http://fibrevillage.com/storage/658-how-to-use-hdfs-fsck-command-to-identify-corrupted-files

Re: datanode + Directory is not writable

mike_bronson7 — Tue, 24 Apr 2018 11:48:44 GMT

what you think about the following steps to fix corrupted files ( I take it from - https://stackoverflow.com/questions/19205057/how-to-fix-corrupt-hdfs-files )

to determine which files are having problems , this ignores lines with nothing but dots and lines talking about replication.

hdfs fsck / | egrep -v '^\.+$' | grep -v eplica

Re: datanode + Directory is not writable

mike_bronson7 — Tue, 24 Apr 2018 11:50:39 GMT

Once you find a file that is corrupt
hdfs fsck /path/to/corrupt/file -locations -blocks -files

Re: datanode + Directory is not writable

Shelton — Tue, 24 Apr 2018 14:19:12 GMT

@Michael Bronson

Above you are trying to fix corrupt HDFS blocks !! With the default replication factor of 3, you should be okay and below is fixing the filesystem

What is your filesystem type ext4 or? You can run

# e2fsck -y /dev/sdc

You will not have an opportunity to validate the corrections being applied. On the other hand if you run

# e2fsck -n /dev/sdc

You can see what would happen without it actually being applied and if you run you'll be asked each

# e2fsck /dev/sdc

time a significant correction needs to be applied.

Re: datanode + Directory is not writable

mike_bronson7 — Tue, 24 Apr 2018 14:55:07 GMT

Dear Geoffrey , the filesystem is ext4

Re: datanode + Directory is not writable

mike_bronson7 — Tue, 24 Apr 2018 15:02:47 GMT

Dear Geoffrey , as you know before performing fsck /dev/sdc , we must umount /grid/sdc , or umount -l /grid/sdc , only then we can run fsck /dev/sdc , so can you approve finally the following steps:

1. umount /grid/sdc or umount -l /grid/sdc in case devise is busy

2. fsck /dev/sdc

Re: datanode + Directory is not writable

Shelton — Thu, 26 Apr 2018 02:35:13 GMT

@Michael Bronson

Any updates?

Re: datanode + Directory is not writable

mike_bronson7 — Thu, 26 Apr 2018 02:38:02 GMT

Hi, Geoffrey

we just waiting for your approval about the following steps:

1. umount /grid/sdc or umount -l /grid/sdc in case devise is busy

2. fsck -y /dev/sdc

3. mount /grid/sdc

Re: datanode + Directory is not writable

Shelton — Thu, 26 Apr 2018 03:03:06 GMT

@Michael Bronson

Avahi is a system which facilitates service discovery on a local network via the mDNS/DNS-SD protocol suite. This enables you to plug your laptop or computer into a network and instantly be able to view other people who you can chat with, find printers to print to or find files being shared. Compatible technology is found in Apple MacOS X (branded Bonjour and sometimes Zeroconf)

The two big benefits of Avahi are name resolution & finding printers, but on a server, in a managed environment, it's of little value.

unmounting and mount filesystems are a common thing especially in Hadoop clusters, your SysOps team should have validated that, but all looks correct to me.

Do a dry run with the below code to see what will be affected that will give you a better picture.

# e2fsck -n /dev/sdc

The data will be reconstructed as you have default replication factor you can later rebalance the HDFS data

Re: datanode + Directory is not writable

mike_bronson7 — Thu, 26 Apr 2018 03:05:19 GMT

yes we already did it on one of the disks , see please - https://community.hortonworks.com/questions/189016/datanode-machine-worker-one-of-the-disks-have-file.html

Re: datanode + Directory is not writable

Shelton — Thu, 26 Apr 2018 03:25:47 GMT

@Michael Bronson

The disk is already unusable go-ahead run fsck with a -y option to repair it 🙂 see above

Either way you will have to replace that dirty disk anyways!