Support Questions

Find answers, ask questions, and share your expertise

What is functional difference between "mapreduce.cluster.temp.dir" and "mapreduce.cluster.local.dir"?

avatar
Rising Star

From, https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de... I understand that "mapreduce.cluster.local.dir" is defined to store intermediate data from mapreduce operation. But, the description of "mapreduce.cluster.temp.dir" was minimal to understand what kind of temporary data is written in there. Would it be cache and temp jars sort of data?

Thank you for your time.

1 ACCEPTED SOLUTION

avatar
Super Guru
@kkanchu

You are reading defaults for MRV1. With YARN/MRV2 mapreduce.cluster.local.dir has been replaced by yarn.nodemanager.local-dirs

This property uses your local disk for storing temporary files. I have not tried mapreduce.cluster.temp.dir but it seems to me the difference is that this is a location in your HDFS and not local file system. You can try running a small sample job and see the difference.

View solution in original post

2 REPLIES 2

avatar
Super Guru
@kkanchu

You are reading defaults for MRV1. With YARN/MRV2 mapreduce.cluster.local.dir has been replaced by yarn.nodemanager.local-dirs

This property uses your local disk for storing temporary files. I have not tried mapreduce.cluster.temp.dir but it seems to me the difference is that this is a location in your HDFS and not local file system. You can try running a small sample job and see the difference.

avatar
Rising Star

@mqureshi thank you for the reply. Yes, "mapreduce.cluster.temp.dir" seems to be legacy property which is no longer available in 2.x.