Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Directory /grid/sdg/hadoop/hdfs/data became unmounted from / . Current mount point:

avatar

we have ambari cluster ( version 2.6 ) with 3 workers machines ( datanode machines )

each worker machine was with 5 disks ( each disk is 20G )

we upgrade the disks to 10 for each worker machine as the following:

1. on each worker we create filesystem for the new 5 disks and perform mount to the partition

2. we update the new disks on dfs.datanode.data.dir , yarn.nodemanager.local-dirs , yarn.nodemanager.log-dirs ( ambari GUI )

3. restart HDFS and YARN

but when we restart the HDFS on the workers ( datanode machines ) we get the following errors:

2017-12-31 19:01:52,509 - 
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
Directory /grid/sdg/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdg . 
Directory /grid/sdh/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdh . 
Directory /grid/sdi/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdi . 
Directory /grid/sdj/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdj . 
Directory /grid/sdk/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdk . Please ensure that mounts are healthy. If the mount change was intentional, you can update the contents of /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist.
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****


so for now all the new disks are empty without hadoop folder under /grid/sdX


please advice how to continue from this stage ?


* we restart teh ambari agent and ambari server without help


Michael-Bronson
1 ACCEPTED SOLUTION

avatar

need to configure the file /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist with all mount point , then start the data-node on each worker machine

Michael-Bronson

View solution in original post

3 REPLIES 3

avatar

need to configure the file /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist with all mount point , then start the data-node on each worker machine

Michael-Bronson

avatar
Rising Star

I have follow what @Michael Bronson indicated,
the folder /var/lib/ambari-agent/data/datanode was not existent. then i created the dfs_data_dir_mount.hist file and copied the content from another server in the cluster

the issue still not solved.

avatar

As you can see, the error message sais to check the file

/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist

This files stores the last mount point for each hdfs folder. In your case, seems that you are trying to mount the HDFS folders on different paths so datanode doesn't start to prevent data loss. Fix the file to point to the new mount points and start the datanode.