Support Questions

Find answers, ask questions, and share your expertise

What is functional difference between "mapreduce.cluster.temp.dir" and "mapreduce.cluster.local.dir"?

Cloudera Employee

From, https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de... I understand that "mapreduce.cluster.local.dir" is defined to store intermediate data from mapreduce operation. But, the description of "mapreduce.cluster.temp.dir" was minimal to understand what kind of temporary data is written in there. Would it be cache and temp jars sort of data?

Thank you for your time.

1 ACCEPTED SOLUTION

Super Guru
@kkanchu

You are reading defaults for MRV1. With YARN/MRV2 mapreduce.cluster.local.dir has been replaced by yarn.nodemanager.local-dirs

This property uses your local disk for storing temporary files. I have not tried mapreduce.cluster.temp.dir but it seems to me the difference is that this is a location in your HDFS and not local file system. You can try running a small sample job and see the difference.

View solution in original post

2 REPLIES 2

Super Guru
@kkanchu

You are reading defaults for MRV1. With YARN/MRV2 mapreduce.cluster.local.dir has been replaced by yarn.nodemanager.local-dirs

This property uses your local disk for storing temporary files. I have not tried mapreduce.cluster.temp.dir but it seems to me the difference is that this is a location in your HDFS and not local file system. You can try running a small sample job and see the difference.

Cloudera Employee

@mqureshi thank you for the reply. Yes, "mapreduce.cluster.temp.dir" seems to be legacy property which is no longer available in 2.x.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.