Directory /grid/sdg/hadoop/hdfs/data became unmounted from / . Current mount point:

[#1]

we have ambari cluster ( version 2.6 ) with 3 workers machines ( datanode machines )

each worker machine was with 5 disks ( each disk is 20G )

we upgrade the disks to 10 for each worker machine as the following:

1. on each worker we create filesystem for the new 5 disks and perform mount to the partition

2. we update the new disks on dfs.datanode.data.dir , yarn.nodemanager.local-dirs , yarn.nodemanager.log-dirs ( ambari GUI )

3. restart HDFS and YARN

but when we restart the HDFS on the workers ( datanode machines ) we get the following errors:

2017-12-31 19:01:52,509 - 
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
Directory /grid/sdg/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdg . 
Directory /grid/sdh/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdh . 
Directory /grid/sdi/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdi . 
Directory /grid/sdj/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdj . 
Directory /grid/sdk/hadoop/hdfs/data became unmounted from / . Current mount point: /grid/sdk . Please ensure that mounts are healthy. If the mount change was intentional, you can update the contents of /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist.
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****

so for now all the new disks are empty without hadoop folder under /grid/sdX

please advice how to continue from this stage ?

* we restart teh ambari agent and ambari server without help

[#2 · Accepted solution]

need to configure the file /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist with all mount point , then start the data-node on each worker machine

[#3]

I have follow what @Michael Bronson indicated,
the folder /var/lib/ambari-agent/data/datanode was not existent. then i created the dfs_data_dir_mount.hist file and copied the content from another server in the cluster

the issue still not solved.

[#4]

As you can see, the error message sais to check the file

/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist

This files stores the last mount point for each hdfs folder. In your case, seems that you are trying to mount the HDFS folders on different paths so datanode doesn't start to prevent data loss. Fix the file to point to the new mount points and start the datanode.