I am a running a insert overwrite query as below.
Insert overwrite directory '/org/data/tmp/webapptempspace/UC3/log' select a.* from a join b on ucase(a.name)=ucase(b.name);
It works fine when one of the table has smaller dataset, but when both tables have huge data, it throws the below error.
Failed with exception Unable to move source /org/data/tmp/webapptempspace/UC3/log/.hive-staging_hive_2016-01-11_04-31-06_067_6297667876520454770-1/-ext-10000 to destination
Yes, I have permission to write. I am able to run the query for a smaller dataset, but not for a larger dataset.
My table 'a' has text format and 'b' has orc format. When I keep both as text, it is working fine.
Is there any more information in the logs? (If you're using HiveServer2, the you can look at those logs, or your client logs if you are using the Hive client directly.) The error is caused by a file system error where it's unable to move the file. The file system should log an error message describing why it failed.
What MoveTask does is moving some files in /tmp volume to /user volume. When the user running doesn't have the right permissions it does not allow moving files between volumes and throws this exception.
-Check /user and /tmp have the full permission
-Check if the below guys are set to true
hive.metastore.server.setugi=true. These parameters instruct hive execute jobs under current shell user
if not then try executing it as root.
I come across the same issue. But when I look at hive logs as suggested by Neeraj, I got to know that there were Space Quota set at my directory level. Once I cleared the quota, I was able to create table even for large data sets.