Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Migrate YARN and HDFS directories

Migrate YARN and HDFS directories

New Contributor

When I first installed Ambari, it detected mount points for /var and /tmp , as well as /hdp . This means that namenode and datanode data is stored in volatile filesystems. How do I remove the /var and /tmp directories from the below config items in Ambari, and migrate the data from those two to /hdp ?

HDFS, Configs, Settings, dfs.namenode.name.dir :

/var/hadoop/hdfs/namenode,/tmp/hadoop/hdfs/namenode,/hdp/hadoop/hdfs/namenode

HDFS, Configs, Settings, dfs.datanode.name.dir :

/var/hadoop/hdfs/data,/tmp/hadoop/hdfs/data,/hdp/hadoop/hdfs/data

YARN, Configs, Advanced, Node Manager, yarn.nodemanager.local-dirs :

/var/hadoop/yarn/local,/tmp/hadoop/yarn/local,/hdp/hadoop/yarn/local

YARN, Configs, Advanced, Node Manager, yarn.nodemanager.log-dirs :

/var/hadoop/yarn/log,/tmp/hadoop/yarn/log,/hdp/hadoop/yarn/log

3 REPLIES 3

Re: Migrate YARN and HDFS directories

Super Guru

@Daniel Skates

If your namenode data is in multiple directories, then it's already in hdp. you can keep the data in hdp and safely delete the other two.

For safety purpose first, make a tar file that contains all your namenode data in case you run into any issue. But you should be fine.

tar -cvf /var/hadoop/hdfs/namenode (as well as with tmp directory)

you should see output that contains ./current/fsimage, ./current/fstime,./current/version, ./current/edits, ./image and ./image/fsimage files

dfs.namenode.name.dir

file://${hadoop.tmp.dir}/dfs/nameDetermines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.

https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Re: Migrate YARN and HDFS directories

New Contributor

How do I know that the complete namenode data is in /hdp , and not partitioned across /var and /tmp ?

Re: Migrate YARN and HDFS directories

New Contributor
Nevermind, from your link

Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.

Don't have an account?
Coming from Hortonworks? Activate your account here