Support Questions
Find answers, ask questions, and share your expertise

Change NameNode and Datanode directories to Azure Data Lake Storage

Change NameNode and Datanode directories to Azure Data Lake Storage

Explorer

I just deployed my cluster on Azure using Cloudbreak. Right now my name node and datanode directories are: /hadoopfs/fs1/hdfs/namenode and /hadoop/hdfs/data,/hadoopfs/fs1/hadoop/hdfs/data,/mnt/resource/hadoop/hdfs/data respectively. I want to change this to a location on my Data Lake Store.
When I just paste the path starting with "adl:/" it throws an error. I setup the access and everything. I just want to change the directories to save space on my local FS.
Also, I am getting alerts like:

Capacity Used:[100%, 28672], Capacity Remaining:[0].

I have 110 GB RAM on my master node. Why is it already used up?

2 REPLIES 2
Highlighted

Re: Change NameNode and Datanode directories to Azure Data Lake Storage

@kskp

Have you set fs.defaultFS to ADLS as well? If so, note that it is not an official setup supported by Hortonworks, and it might cause HDFS calculating free space incorrectly.

Hope this helps!

Highlighted

Re: Change NameNode and Datanode directories to Azure Data Lake Storage

Explorer

I did. I just had to change the value of Reserved Space for HDFS in HDFS configs. Thanks for commenting.

Don't have an account?