Hi, "HiveServer2 Enable Impersonation is setting to TRUE" is probably the reason. When Impersonation is true, it means Hive will impersonate as the end user who runs the query to submit jobs. Your ACL output showed that the directory is owned by "hive:hive" and as @Tomas79 found out, you have sticky bit set, so if hive needs to impersonate as the end user, the end user who runs the query will not be able to delete the path as he/she is not the owner. If impersonation is OFF, then HS2 will run query as "hive" user (the user that runs HS2 process), then you should not see such issue. I assume you have no sentry? As sentry will require Impersonation to be OFF on HS2 side, so that all queries will be running under "hive" user. To test the theory, try to remove the sticky bit on this path and drop again in Hive. Cheers Eric ... View more

You can use a script like this to create snapshots of old and new files - i.e. search files which are older than 3 days and search for files which are newer than 3 days, just make sure, you use the correct path to the cloudera jars. In the case of CDH5.15:   #!/bin/bash now=`date +"%Y-%m-%dT%H:%M:%S"` hdfs dfs -rm /data/cleanup_report/part=older3days/* hdfs dfs -rm /data/cleanup_report/part=newer3days/* hadoop jar /opt/cloudera/parcels/CDH/jars/search-mr-1.0.0-cdh5.15.1.jar org.apache.solr.hadoop.HdfsFindTool -find /data -type d -mtime +3 | sed "s/^/${now}\tolder3days\t/" | hadoop fs -put - /data/cleanup_report/part=older3days/data.csv hadoop jar /opt/cloudera/parcels/CDH/jars/search-mr-1.0.0-cdh5.15.1.jar org.apache.solr.hadoop.HdfsFindTool -find /data -type d -mtime -3 | sed "s/^/${now}\tnewer3days\t/" | hadoop fs -put - /data/cleanup_report/part=newer3days/data.csv   Then create an external table with partitions on top of this HDFS folder. ... View more

Yes Spark2-shell and spark2-submit has these issues as well. Any insights ... View more

Could you let me know what is the issue you are facing ? whats the error ?  ... View more

Thankyou Ben .. ... View more

No it was just one insert and after the repeat it succeeded, so I am not able to reproduce, and thus no patterns. CDH 5.15 Can you give me a detailed hint how to get the full stacktrace (from the Impala daemon?) of the failed fragment? I dont have the query profile (already deleted) but as I can remember one of the fragment (out of 10) was waiting for almost 2h to HDFS sink, others finished within a minute. Maybe it is a hdfs issue? ... View more

Could you supply what your /var/log/cloudera-scm-agent/certmanager.log looked like after the successful installation? For comparative purposes? ... View more

Thank you all for your help ... View more