question What is functional difference between "mapreduce.cluster.temp.dir" and "mapreduce.cluster.local.dir"? in Archives of Support Questions (Read Only)

What is functional difference between "mapreduce.cluster.temp.dir" and "mapreduce.cluster.local.dir"?

kkanchu — Fri, 31 Mar 2017 00:13:08 GMT

From, https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml I understand that "mapreduce.cluster.local.dir" is defined to store intermediate data from mapreduce operation. But, the description of "mapreduce.cluster.temp.dir" was minimal to understand what kind of temporary data is written in there. Would it be cache and temp jars sort of data?

Thank you for your time.

Re: What is functional difference between "mapreduce.cluster.temp.dir" and "mapreduce.cluster.local.dir"?

mqureshi — Fri, 31 Mar 2017 04:32:58 GMT

@kkanchu

You are reading defaults for MRV1. With YARN/MRV2 mapreduce.cluster.local.dir has been replaced by yarn.nodemanager.local-dirs

This property uses your local disk for storing temporary files. I have not tried mapreduce.cluster.temp.dir but it seems to me the difference is that this is a location in your HDFS and not local file system. You can try running a small sample job and see the difference.

Re: What is functional difference between "mapreduce.cluster.temp.dir" and "mapreduce.cluster.local.dir"?

kkanchu — Fri, 31 Mar 2017 04:51:28 GMT

@mqureshi thank you for the reply. Yes, "mapreduce.cluster.temp.dir" seems to be legacy property which is no longer available in 2.x.