In that case, you could try "refresh <partition>" and see the peak JVM memory usage on the Catalogds and the Impalads and if it is close to hitting OOM, increase the -Xmx [1].   Also, from our experience, using incremental stats with high refresh load can quickly trigger OOM issues. So, better not to rely on them.   [1] https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Catalogd-OutOfMemoryError-Java-heap-space/td-p/41177 ... View more

- In general, the suggestion is to run as few 'refresh'es as possible. So I'd suggest running a 'refresh <table>' every 20 mins than running a 'refresh <partition>' every 2 mins if that meets your application SLAs.   - Each refresh triggers a spike in working memory on the Catalog (and the Impalads) due to thrift serialization (and deserialization on the Impalads) and the cost is constant irrespective of whether you run a refresh on the table or the partition since we serialize (and deserialize) the whole table object. This can be costly if the partitioned table is huge or with lots of files and blocks. (https://issues.apache.org/jira/browse/IMPALA-3127)   - FWIW, these load operations are much faster (~2x on secure and ~5x on insecure) starting CDH5.14 due to performance enhancements. ... View more

Thanks for checking. The kvno.s and principals look fine to me.   - Can you confirm that the OS and kerberos client libs are same on all these nodes or are they different? (lsb_release -a, rpm -qa | grep krb...) - Are you able to run any other services on 02 and 04 like datanode etc? Is the issue only with Impala? ... View more

Can you please check the kvno. of the principals in the failing hosts match with the kvno. of the principal in the KDC? Also, do you see any difference in the output of "klist -kt <path_to_impala.keytab>" on working and non-working hosts? especially in the KVNO. section? In the above pasted output, I only see it for non-working hosts where its 1, Is it the same for working hosts too? ... View more

Sorry for the inconvenience here. I think the errors messages should've been better to aid debugging. Can you paste the contents of the catalog-server startup till it fails to obtain tgt? Also did you try manually kinit'ing with the keytab and catalog principal and make sure it works? Whats the output of "klist -kt /etc/impala/conf/impala.keytab" ?   ... View more

Yep, try using the TCompactProtocol to deserialize (Default is TBinaryProtocol). Initialize the deserializer like,   new TDeserializer(TCompactProtocol.Factory())     ... View more

Sorry I missed this. Impala prepends the query hash to the base64 string while logging to the file. In your case that is 4c4d52afea4a40a1:802034563d1cfc93. Just remove it from dataCompressed string and it should work. ... View more

What version of sentry are you running? Do you have some stray jars somewhere in the classpath that Catalog can be picking up? ... View more