Member since
12-15-2020
1
Post
0
Kudos Received
0
Solutions
12-15-2020
10:34 AM
Hello, We have exactly the same trouble. At first, be sure that Insert Overwite does not create the folder of your Hive table. Your folder has been created during the "CREATE TABLE..... LOCATION 'hdfs://titan/dev/10112/app/TC30/dataiku/CONFIG_ANOTHER_TEST/output'...." In fact, you are using an old version of DSS (Data Science Studio) from Dataiku. For us, we are using the 5.0.3 DSS version, knowing that at this time, 8.0.4 is the last release. As you said : " the vendor application first delete the folder then do the insert overwrite on the external table". That's true. But you have to know that Hive commands, like INSERT OVERWRITE or ANALYZE TABLE...COMPUTE STATISTICS, create a temporary folder. You have to look at this line of your log : Loading data to table dev_tc30_dataiku.config_another_test_output from hdfs://titan/tmp/.hive-staging_hive_2018-11-21_10-45-41_452_43360044430205414-24417/-ext-10000 An .hive_staging..... temporary folder has been created under /tmp directory. Please go into Ambari, and look at this Hive parameter hive.exec.stagingdir I am sure that this actual value is /tmp/.hive_staging Please modify this value into default value .hive_staging, and test again using DSS. Normally, Hive will now create a temporary folder under hdfs://titan/dev/10112/app/TC30/dataiku/CONFIG_ANOTHER_TEST/output/.hive-staging........ Of course, this .hive-staging... directory is temporary, but it is important to notice that the output folder has been recreated, and so the Insert Overwrite will be Ok. For us, we have the same trouble because we did an upgrade of our HDP cluster from 2.6.1 into 2.6.5 release; and we have to rollback the hive.exec.stagingdir parameter from /tmp/.hive_staging into .hive_staging Your ticket has been created in 11-21-2018, so at this date, the latest release of DSS was 5.0.3 Please notice that in next releases, Dataiku has modified this behaviour, and I don't think that now the Hive folder is always deleted. Regards, Gilles
... View more