Created 08-10-2016 04:42 AM
When I first installed Ambari, it detected mount points for /var and /tmp , as well as /hdp . This means that namenode and datanode data is stored in volatile filesystems. How do I remove the /var and /tmp directories from the below config items in Ambari, and migrate the data from those two to /hdp ?
HDFS, Configs, Settings, dfs.namenode.name.dir :
/var/hadoop/hdfs/namenode,/tmp/hadoop/hdfs/namenode,/hdp/hadoop/hdfs/namenode
HDFS, Configs, Settings, dfs.datanode.name.dir :
/var/hadoop/hdfs/data,/tmp/hadoop/hdfs/data,/hdp/hadoop/hdfs/data
YARN, Configs, Advanced, Node Manager, yarn.nodemanager.local-dirs :
/var/hadoop/yarn/local,/tmp/hadoop/yarn/local,/hdp/hadoop/yarn/local
YARN, Configs, Advanced, Node Manager, yarn.nodemanager.log-dirs :
/var/hadoop/yarn/log,/tmp/hadoop/yarn/log,/hdp/hadoop/yarn/log
Created 08-10-2016 05:02 AM
If your namenode data is in multiple directories, then it's already in hdp. you can keep the data in hdp and safely delete the other two.
For safety purpose first, make a tar file that contains all your namenode data in case you run into any issue. But you should be fine.
tar -cvf /var/hadoop/hdfs/namenode (as well as with tmp directory)
you should see output that contains ./current/fsimage, ./current/fstime,./current/version, ./current/edits, ./image and ./image/fsimage files
dfs.namenode.name.dir | file://${hadoop.tmp.dir}/dfs/name | Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. |
https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Created 08-10-2016 05:05 AM
How do I know that the complete namenode data is in /hdp , and not partitioned across /var and /tmp ?
Created 08-10-2016 05:10 AM
Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.