Thanks Alex.   I'll read up on this - but I think I would re-echo my prior sentiment... we've taken IMHO a huge step backwards here in terms of scalability.   I have hundreds of tables with thousands of partitions.   Compute incremental stats used to work BEAUTIFULLY. All of our code is set to use it, and was exponentially better than trying to do a full compute stats run. Net: We used to push a single button, and everything would just work.   Now however, we have to look at programatically trying to compute a bunch of statistics - and then manually update things... for hundreds of partitions, and thousands of tables. Net: We now have to build, and then manually push tens of thousands of buttons instead of just one. And then maybe it changes again in the next release. This is a huge, step backwards IMHO.   Just making that flag a variable or a configuration option at least sounds like a very easy. 🙂   Don't mean to sound like I'm complaining. But we had an amazing, wonderful ice cream cone that was fantastic - and someone just came and slapped it out of our hands. 🙂 ... View more

This issue is absolutely crippling us. We've had to stop computing stats on virtually all of our tables, because it simply takes far too long to do so on the entire table - and we get this error now for almost all incremental updates as well.   This effectively makes impala unusable at scale.   If there's any potential work around (sans dowgrading cluster) it would be greatly appreciated. ... View more

I had tried the environmental variable approach as well when I originally posted this - didn't work for me either.   Having to go in and delete the file from the cluster definitely does not sound like an attractive approach. ... View more

Thanks had tried that a few increments, but it didn't seem to help. Upped it up again, and now things do appear better.   CDH 5.1 we had it set to 1GB, never had this issue.   CDH 5.2 it's now set to 4GB, and so far it's looking OK. 1-3GB exhibited the issue.   4X the memory between versions seems high but hopefully that keeps it happy.   Thanks ... View more

Not sure if it helps your situation, but I noticed a few bug reports on HIVE direct SQL causing issues with table statistics/etc. Per the workaround described in those tickets, I disabled:   hive.metastore.try.direct.sql   And in parallel also disabled the Sentry service and went back to using the static file.   Although the bug reports aren't directly releated to slowness.... it appears most of our Hive issues are looking a lot better. Queries that were having issues are now working, those that were slow are fast again, etc. Still having some weird issues with Impala.   Overall pretty dissapointed with the 5.2 upgrade. Looks like a lot of bugs, and what was pitched as a big performance increase appears to be anything but. Hopefully 5.2.1 is an improvement... ... View more

Noticed the same thing - for both Hive and Impala. Don't see anything obviously wrong.   From what I can tell so far, it appears to have something to do with the Hive Metastore. It occasionally fails the Canary tests, and trying to drop tables with large partitions throws errors like:   FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:org.datanucleus.exceptions.NucleusDataStoreException: Update of object "org.apache.hadoop.hive.metastore.model.MStorageDescriptor@2e8e6344" using statement "UPDATE `SDS` SET `CD_ID`=? WHERE `SD_ID`=?" failed : java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction   From Impala we see a lot of tables not having their Metadata loaded:   Missing tables were not received in 120000ms. Load request will be retried. Requesting prioritized load of table(s): .... repeats ... repeats ... and then usually eventually loads.   The net result is a query that used to take X amount of time now takes significantly longer. ... View more