In general I configure disk space allocation for tez job spills the same way as Yarn intermediate data. Please find here some discussions regarding how to configure it: http://community.hortonworks.com/questions/2230/recommended-size-for-yarnnodemanagerresourcelocal.html#answer-2282 http://community.hortonworks.com/questions/1405/can-you-please-advise-about-how-best-to-use-this-s.html?redirectedFrom=1711 ... View more

If you use the same partitions for yarn intermediate data than for the HDFS blocks, then you might also consider setting the fs.datanode.du.reserved property, which reserves some space on those partitions for non-hdfs use (such as intermediate yarn data). One base recommendation I saw on my first Hadoop training long time ago was to dedicate 25% of the "data disks" for that kind of intermediate data. I guess the optimal answer should consider the maximum amount of intermediate data you can get at the same time (when launching a job, do you use all the data of HDFS as input data?) and dedicate the space for yarn.nodemanager.resource.local-dirs accordingly. I would also recommend turning on the property mapreduce.map.output.compress in order to reduce the size of the intermediate data. ... View more

One thing I did in the past to avoid that script to be loaded multiple times was editing that script and add a kind of global variable to detect if the script needs to be executed again. Something like that (put that at the beginning of the script, just after the first line #!/bin/bash): [ "x$SCRIPT_HADOOP_ENV_LOADED" = "x1" ] && return export SCRIPT_HADOOP_ENV_LOADED=1 ... View more