question Re: what is the default behavior of insert overwrite on external hdfs table? in Support Questions

question Re: what is the default behavior of insert overwrite on external hdfs table? in Support Questions https://community.cloudera.com/t5/Support-Questions/what-is-the-default-behavior-of-insert-overwrite-on-external/m-p/307719#M223338 Hello,We have exactly the same trouble.At first, be sure that Insert Overwite does not create the folder of your Hive table.Your folder has been created during the "CREATE TABLE..... LOCATION 'hdfs://titan/dev/10112/app/TC30/dataiku/CONFIG_ANOTHER_TEST/output'...." In fact, you are using an old version of DSS (Data Science Studio) from Dataiku. For us, we are using the 5.0.3 DSS version, knowing that at this time, 8.0.4 is the last release. As you said : " the vendor application first delete the folder then do the insert overwrite on the external table". That's true. But you have to know that Hive commands, like INSERT OVERWRITE or ANALYZE TABLE...COMPUTE STATISTICS, create a temporary folder. You have to look at this line of your log :Loading data to table dev_tc30_dataiku.config_another_test_output from hdfs://titan/tmp/.hive-staging_hive_2018-11-21_10-45-41_452_43360044430205414-24417/-ext-10000 An .hive_staging..... temporary folder has been created under /tmp directory. Please go into Ambari, and look at this Hive parameter hive.exec.stagingdirI am sure that this actual value is /tmp/.hive_stagingPlease modify this value into default value .hive_staging, and test again using DSS. Normally, Hive will now create a temporary folder under hdfs://titan/dev/10112/app/TC30/dataiku/CONFIG_ANOTHER_TEST/output/.hive-staging........ Of course, this .hive-staging... directory is temporary, but it is important to notice that the output folder has been recreated, and so the Insert Overwrite will be Ok. For us, we have the same trouble because we did an upgrade of our HDP cluster from 2.6.1 into 2.6.5 release; and we have to rollback the hive.exec.stagingdir parameter from /tmp/.hive_staging into .hive_staging Your ticket has been created in 11-21-2018, so at this date, the latest release of DSS was 5.0.3Please notice that in next releases, Dataiku has modified this behaviour, and I don't think that now the Hive folder is always deleted.Regards,Gilles     Tue, 15 Dec 2020 18:34:54 GMT rouardg 2020-12-15T18:34:54Z