Hive Scratch Directory

[#1]

My cluster is having KMS so when ever I am running the load or manipulate in hive getting error saying that intermediate files can't be moved from non EZ to EZ so I did scratchdir set in the Hive shell still it is taking /tmp/hive/....instead of taking the latest scratch dir... As a work around I did while launching the hive shell if i give hive --hiveconf hive.exec.scratchdir then it is working fine..

Please suggest what was the issue ..I am using hive 0.14.0.2.2.4.10-3

[#2 · Accepted solution]

@ammu ch

Can you share the output of this command?

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p ******* get <hivehosts> <cluster> hive-site |grep dir

And

sudo -u hdfs hdfs crypto -listZones|grep <usename>

[#3]

@ammu ch Ok I got the version, but still as per the fix it should use same EZ directory for scratch where table exist. https://issues.apache.org/jira/browse/HIVE-8065

[#4]

Hope you got my point this is basicaly i am having multiple EZ so each time based on the table I have to change the scratch directory ..within Hive CLI if I set scratch dir it is not taking that as temp directory ...so every time I have to exit and start hive shell with new scratchdir as --hiveconf ...

[#5]

Try creating .hiverc file inside the directory from where you are trying to run hive shell and add below line inside the file and run hive shell. Thats the only way remaining or use latest hive release. Thank You.

SET hive.exec.scratchdir=/TMP/dir;

[#6]

🙂 not sure what soultion you are suggesting ..instead of exit and restart the hive CLI command ..again everytimteI have to edit this file to get new scratchdir and restart the Hive CLI..

[#7]

Sorry no other solution from my side since hive doesn't allow few parameters change after initiating session and this is how it works. Or Wait for other experts to answer.

[#8]

we can change the parameter in session level by making the entry in hive-site.xml.

example:

for property hive.exec.scratchdir

hive-site.xml

==ADD

<property>

<name>hive.security.authorization.sqlstd.confwhitelist.append</name>

<value>hive\.exec\.scratchdir|</value>

<description>append conf property in white list followed by pipeline</description>

</property>

==

Restart metastore and hiveserver

beeline> set hive.exec.scratchdir=/user/hive/scratch;

beeline> set hive.exec.scratchdir;

Hope it helps, if someone is looking for the same.

Thanks,

Manjil