Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to change the query logs on the /tmp/user/hive.log in CDH 6.3

avatar
Rising Star

hi, my partition in /tmp is almost in full capacity, that is why some of my jobs are failing due to space issue, how can I change the default location of the hive query logs of a user? currently it is in the /tmp/user/hive.log. but I want to change it to /data/tmp.

 

I've tried changing the value of hive.querylog.location in the /etc/hive/conf.cloudera.hive/hive-site.xml in all of servers but still no luck.

Screen Shot 2020-06-29 at 1.52.07 PM.png

4 REPLIES 4

avatar
Master Collaborator

@Mondi 

 

You can set the hive scratch directories to modify the intermediate data storage location from default /tmp

hive.exec.local.scratchdir- 

Local Directory where Hive stores jars and data when performing a MapJoin optimization. If not specified, Hive uses a default location.

hive.exec.scratchdir

Directory in HDFS where Hive writes intermediate data between MapReduce jobs. If not specified, Hive uses a default location.

 

Refer

https://docs.cloudera.com/documentation/enterprise/6/properties/6.3/topics/cm_props_cdh600_hive.html...

 

Hope this helps,
Paras
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

 

avatar
Rising Star

Hi @paras I have already changed the properties you've said but still the hive.log is still creating inside the /tmp directory.

Screen Shot 2020-06-30 at 6.25.34 PM.png

 

But if you check my hive config, this is already been edited:

Screen Shot 2020-06-30 at 6.28.45 PM.png

avatar
Master Collaborator

@Mondi 

 

What are these properties set to in your yarn configurations?

'yarn.app.mapreduce.am.staging-dir'

'yarn.app.mapreduce.am.staging-dir' 

 

Also, can you try running below query before your hive query and share results

set hive.exec.scratchdir=/tmp/mydir;

 

Regards,

Paras

avatar
Rising Star

Hi @paras 

 

here is the value for yarn.app.mapreduce.am.staging-dir'

 

Screen Shot 2020-07-02 at 11.30.30 AM.png

 

also i think i don't longer need to set this set hive.exec.scratchdir=/tmp/mydir because i've already set in the hive config,

Screen Shot 2020-07-02 at 11.32.05 AM.png