Support Questions
Find answers, ask questions, and share your expertise

Prevent HDFS from creating data directories

Solved Go to solution

Prevent HDFS from creating data directories

Explorer

Hello community,

I am using HDP 3.0 and I have configured HDFS DataNode Directories as follow: /grid0/hadoop/hdfs/data,/grid1/hadoop/hdfs/data,/grid/hadoop/hdfs/data/

One of my hosts (lets call him hostA) has only one mount point (/grid) and the other host (hostB) has 2 mount points (/grid0 & /grid1)

HDFS automatically creates the directories grid0 and grid1 in "hostA" and grid in "hostB".

Is there any way to prevent HDFS from doing that?

Thanks in advance

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Prevent HDFS from creating data directories

@Christos Stefanopoulos,

That is the expected behaviour. If you want to achieve this, then you need to create separate configs for each host.

For ex: hostA will have only /grid in datanode dirs and hostB will have /grid0 and /grid1 in their datanode dirs.

You can do that using Ambari config groups

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/using_host_con...

View solution in original post

2 REPLIES 2

Re: Prevent HDFS from creating data directories

@Christos Stefanopoulos,

That is the expected behaviour. If you want to achieve this, then you need to create separate configs for each host.

For ex: hostA will have only /grid in datanode dirs and hostB will have /grid0 and /grid1 in their datanode dirs.

You can do that using Ambari config groups

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/using_host_con...

View solution in original post

Re: Prevent HDFS from creating data directories

Explorer

@Aditya Sirna

Thank you so much for your help, this solved my problem.