question Re: How do I configure HDFS storage types and policies? in Archives of Support Questions (Read Only)

How do I configure HDFS storage types and policies?

sbradshaw — Tue, 27 Oct 2015 00:24:56 GMT

According to the Apache and our own documentation, I would use the hdfs dfsadmin -setStoragePolicy and -getStoragePolicy commands to configure and use HDFS storage types and policies. However, on my HDP 2.3.0 cluster, installed using Ambari 2.1.1, the hdfs dfsadmin command does not have the -getStoragePolicy and -setStoragPolicy commands. So I do I configure storage types and policies?

Re: How do I configure HDFS storage types and policies?

jstraub — Tue, 27 Oct 2015 00:28:47 GMT

I think the command has changed, its not hdfs dfsadmin anymore

Try this:

Set a storage policy to a file or a directory.

hdfs storagepolicies -setStoragePolicy -path <path> -policy <policy>

Get the storage policy of a file or a directory.

hdfs storagepolicies -getStoragePolicy -path <path>

Source: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html#Set_Storage_Policy

Re: How do I configure HDFS storage types and policies?

Kkalita — Mon, 19 Aug 2019 12:55:09 GMT

Following steps working for me:

1.Create mount points:

#mkdir /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3

#chown hdfs:hadoop /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3

(**We are using the configuration for test purpose only, so no disks are mounted.)

2.Login to Ambari > HDFS>setting
3.Add datanode directories as shown below:
Datanode>datanode directories:
[DISK]/hadoop/hdfs/data,[SSD]/hadoop/hdfs/data1,[RAMDISK]/hadoop/hdfs/data2,[ARCHIVE]/hadoop/hdfs/data3
Restart hdfs hdfs service.
Restart all other afftected services.
Create a directory /cold
# su hdfs
[hdfs@hdp-qa2-n1 ~]$ hadoop fs -mkdir /cold
Set COLD storage policy on /cold
[hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -setStoragePolicy -path /cold -policy COLD
Set storage policy COLD on /cold
5. Run get storage policy:
[hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -getStoragePolicy -path /cold
The storage policy of /cold:
BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}