Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Tiered Storage RAM_DISK

Tiered Storage RAM_DISK

I am following:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_hdfs-administration/content/configure_me...

I've been able to set it up for SSD but can't get RAM_DISK to work. I can create the ramdisk at the linux level, attach a storage policy and write to it outside of Hadoop. It appears that HDP set’s up its dir structure on the ramdisk but all write attempts end up on regular [DISK]. When I attempt to specify the dfs.datanode.data.dir using the URI (i.e [RAM_DISK]file:///ramdisk/hdfs) as detailed in the doc mentioned above,the datanode fails to start, so I use [RAM_DISK]/ramdisk/hdfs.

The document mentioned above seems to have some inconsistencies:

a) Says to use dfs.data.dir not dfs.datanode.data.dir

b) Says to set dfs.checksum.type to NULL.

When the datanode is starting a warning message pops up that says the checksum type is invalid and is reverting to CRC.

Anyone with some experience to share on TIERED STORAGE?

1 REPLY 1
Highlighted

Re: Tiered Storage RAM_DISK

Rising Star

dfs.data.dir is deprecated by dfs.datanode.data.dir in new version of Hadoop. Both should work in HDP 2.6.1. Please check and ensure entries are comma separated without any extra space.

bq. "When I attempt to specify the dfs.datanode.data.dir using the URI (i.e [RAM_DISK]file:///ramdisk/hdfs) as detailed in the doc mentioned above,the datanode fails to start."

@Constantin Stanca, can you post the datanode log when datanode failed to start?

Don't have an account?
Coming from Hortonworks? Activate your account here