Created on 06-28-2020 10:53 PM - edited 09-16-2022 07:37 AM
hi, my partition in /tmp is almost in full capacity, that is why some of my jobs are failing due to space issue, how can I change the default location of the hive query logs of a user? currently it is in the /tmp/user/hive.log. but I want to change it to /data/tmp.
I've tried changing the value of hive.querylog.location in the /etc/hive/conf.cloudera.hive/hive-site.xml in all of servers but still no luck.
Created 06-29-2020 03:15 AM
You can set the hive scratch directories to modify the intermediate data storage location from default /tmp
hive.exec.local.scratchdir-
Local Directory where Hive stores jars and data when performing a MapJoin optimization. If not specified, Hive uses a default location. |
hive.exec.scratchdir
Directory in HDFS where Hive writes intermediate data between MapReduce jobs. If not specified, Hive uses a default location. |
Refer
Hope this helps,
Paras
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Created 06-30-2020 03:29 AM
Hi @paras I have already changed the properties you've said but still the hive.log is still creating inside the /tmp directory.
But if you check my hive config, this is already been edited:
Created 06-30-2020 05:50 AM
What are these properties set to in your yarn configurations?
'yarn.app.mapreduce.am.staging-dir'
'yarn.app.mapreduce.am.staging-dir'
Also, can you try running below query before your hive query and share results
> set hive.exec.scratchdir=/tmp/mydir;
Regards,
Paras
Created 07-01-2020 08:32 PM
Hi @paras
here is the value for yarn.app.mapreduce.am.staging-dir'
also i think i don't longer need to set this set hive.exec.scratchdir=/tmp/mydir because i've already set in the hive config,