question Re: Hive Scratch Directory in Support Questions

Hive Scratch Directory

Sreedhar_ch — Thu, 16 Jun 2016 01:23:34 GMT

My cluster is having KMS so when ever I am running the load or manipulate in hive getting error saying that intermediate files can't be moved from non EZ to EZ so I did scratchdir set in the Hive shell still it is taking /tmp/hive/....instead of taking the latest scratch dir... As a work around I did while launching the hive shell if i give hive --hiveconf hive.exec.scratchdir then it is working fine..

Please suggest what was the issue ..I am using hive 0.14.0.2.2.4.10-3

Re: Hive Scratch Directory

jyadav — Thu, 16 Jun 2016 17:15:35 GMT

@ammu ch

Can you share the output of this command?

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p ******* get <hivehosts> <cluster> hive-site |grep dir

And

sudo -u hdfs hdfs crypto -listZones|grep <usename>

Re: Hive Scratch Directory

jyadav — Thu, 16 Jun 2016 17:20:17 GMT

@ammu ch Ok I got the version, but still as per the fix it should use same EZ directory for scratch where table exist. https://issues.apache.org/jira/browse/HIVE-8065

Re: Hive Scratch Directory

Sreedhar_ch — Thu, 16 Jun 2016 17:50:26 GMT

Hope you got my point this is basicaly i am having multiple EZ so each time based on the table I have to change the scratch directory ..within Hive CLI if I set scratch dir it is not taking that as temp directory ...so every time I have to exit and start hive shell with new scratchdir as --hiveconf ...

Re: Hive Scratch Directory

jyadav — Thu, 16 Jun 2016 18:02:15 GMT

Try creating .hiverc file inside the directory from where you are trying to run hive shell and add below line inside the file and run hive shell. Thats the only way remaining or use latest hive release. Thank You.

SET hive.exec.scratchdir=/TMP/dir;

Re: Hive Scratch Directory

Sreedhar_ch — Thu, 16 Jun 2016 18:09:34 GMT

🙂 not sure what soultion you are suggesting ..instead of exit and restart the hive CLI command ..again everytimteI have to edit this file to get new scratchdir and restart the Hive CLI..

Re: Hive Scratch Directory

jyadav — Thu, 16 Jun 2016 18:18:45 GMT

Sorry no other solution from my side since hive doesn't allow few parameters change after initiating session and this is how it works. Or Wait for other experts to answer.

Re: Hive Scratch Directory

manjilhk — Tue, 21 Aug 2018 15:37:41 GMT

we can change the parameter in session level by making the entry in hive-site.xml.

example:

for property hive.exec.scratchdir

hive-site.xml

==ADD

<name>hive.security.authorization.sqlstd.confwhitelist.append</name>

<value>hive\.exec\.scratchdir|</value>

<description>append conf property in white list followed by pipeline</description>

</property>

Restart metastore and hiveserver

beeline> set hive.exec.scratchdir=/user/hive/scratch;

beeline> set hive.exec.scratchdir;

Hope it helps, if someone is looking for the same.

Thanks,

Manjil