In that case, you could try "refresh <partition>" and see the peak JVM memory usage on the Catalogds and the Impalads and if it is close to hitting OOM, increase the -Xmx [1].   Also, from our experience, using incremental stats with high refresh load can quickly trigger OOM issues. So, better not to rely on them.   [1] https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Catalogd-OutOfMemoryError-Java-heap-space/td-p/41177 ... View more

- In general, the suggestion is to run as few 'refresh'es as possible. So I'd suggest running a 'refresh <table>' every 20 mins than running a 'refresh <partition>' every 2 mins if that meets your application SLAs.   - Each refresh triggers a spike in working memory on the Catalog (and the Impalads) due to thrift serialization (and deserialization on the Impalads) and the cost is constant irrespective of whether you run a refresh on the table or the partition since we serialize (and deserialize) the whole table object. This can be costly if the partitioned table is huge or with lots of files and blocks. (https://issues.apache.org/jira/browse/IMPALA-3127)   - FWIW, these load operations are much faster (~2x on secure and ~5x on insecure) starting CDH5.14 due to performance enhancements. ... View more

@BellRizz Adding to Tim's comment, we have seen this warning pop up if there are some issues with Catalog server connecting to datanodes to get block locations (either DNs are busy/not -responding for some reason etc.). This usually goes away when the load on DN is low. Might be worth checking this scenario as well (Usually something is logged into the Catalog server logs around the time this error occurs).   The later versions of Impala (shipped with CDH 5.12 and later) has a new way of fetching these block locations and has a lower likelyhood of this warning.  ... View more

Yes the persistent UDFs feature is included in 5.8.0. Also, Java UDFs require a special syntax  to make them persistent.  You can find examples here.   https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_create_function.html#create_function   Regular way of creating Impala Java UDFs, will wipe them off from the Catalog after a restart. This is done so to be consistent with Hive. ... View more

Do you have some comments in your script right before the insert stmt? If yes, you could be running into a known bug https://issues.cloudera.org/browse/IMPALA-2195 where we incorrectly print "Fetched" rows instead of "Inserted" rows as the parser doesn't detect its an insert stmt. ... View more

Oh, I didn't know that. May be a bug like Skye mentioned in the other thread. Thanks for looking it up. ... View more

I don't think (1) is possible but you can set "--insert_inherit_permissions=true" for (2). Can you give that a try? ... View more

Thanks for checking. The kvno.s and principals look fine to me.   - Can you confirm that the OS and kerberos client libs are same on all these nodes or are they different? (lsb_release -a, rpm -qa | grep krb...) - Are you able to run any other services on 02 and 04 like datanode etc? Is the issue only with Impala? ... View more

Can you please check the kvno. of the principals in the failing hosts match with the kvno. of the principal in the KDC? Also, do you see any difference in the output of "klist -kt <path_to_impala.keytab>" on working and non-working hosts? especially in the KVNO. section? In the above pasted output, I only see it for non-working hosts where its 1, Is it the same for working hosts too? ... View more

Sorry for the inconvenience here. I think the errors messages should've been better to aid debugging. Can you paste the contents of the catalog-server startup till it fails to obtain tgt? Also did you try manually kinit'ing with the keytab and catalog principal and make sure it works? Whats the output of "klist -kt  /etc/impala/conf/impala.keytab" ?   ... View more

Thanks for clarifying. How are you creating the table? Can you paste the sql here? ... View more

Does the URI contain capital case letters? If yes, it could be IMPALA-2695, fixed in CDH 5.5.2. ... View more

This issue is being tracked at IMPALA-1822. We don't have a fix for it right now. May be you can workaround this using a cron job? ... View more

You are most likely hitting  IMPALA-2154. The first gz file might have had multiple streams and when you repacked it, it became a single streamed one. We are working on a fix for this issue currently. Please follow the jira for updates. ... View more