Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How do I configure HDFS storage types and policies?

avatar
New Contributor

According to the Apache and our own documentation, I would use the hdfs dfsadmin -setStoragePolicy and -getStoragePolicy commands to configure and use HDFS storage types and policies. However, on my HDP 2.3.0 cluster, installed using Ambari 2.1.1, the hdfs dfsadmin command does not have the -getStoragePolicy and -setStoragPolicy commands. So I do I configure storage types and policies?

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Rising Star

Following steps working for me:

  • 1.Create mount points:

#mkdir /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3

#chown hdfs:hadoop /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3

(**We are using the configuration for test purpose only, so no disks are mounted.)

  • 2.Login to Ambari > HDFS>setting
  • 3.Add datanode directories as shown below:
  • Datanode>datanode directories:
  • [DISK]/hadoop/hdfs/data,[SSD]/hadoop/hdfs/data1,[RAMDISK]/hadoop/hdfs/data2,[ARCHIVE]/hadoop/hdfs/data3
  • 5050-sp.png

    Restart hdfs hdfs service.

    Restart all other afftected services.

    Create a directory /cold

    # su hdfs

    [hdfs@hdp-qa2-n1 ~]$ hadoop fs -mkdir /cold

    Set COLD storage policy on /cold

    [hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -setStoragePolicy -path /cold -policy COLD

    Set storage policy COLD on /cold

    5. Run get storage policy:

    [hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -getStoragePolicy -path /cold

    The storage policy of /cold:

    BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}