Thanks for your reply, but it didn't work. I tried using other functions but it brings up the date time without milliseconds.  But I want the milliseconds as well. ... View more

It looks like there was plenty of memory available in the system, that query just hit its individual memory limit.   There were a lot of improvements to avoid out-of-memory between 5.15 and 6.1, particularly for queries with a lot of scans that use a significant amount of memory. It looks like one of the scans was using a large chunk of the query memory:   HDFS_SCAN_NODE (id=0): Total=393.15 MB Peak=393.15 MB   There's one specific regression that I'm aware of that affected Avro scans: https://issues.apache.org/jira/browse/IMPALA-7078. The fix is in 5.15.1 and 5.15.2. I don't know the file format but thought I'd flag that. The IMPALA-7078 fix actually had a few tweaks that would benefit all file formats too.   So I'd suggest: Give the queries a bit more memory - in practice we've seen 2GB be a lot better with a wider variety of queries in CDH5.x. 1GB is a bit squeezy for a query with 49 operators. Pick up the 5.15.2 or 5.16.2 maintenance releases to get the fix for IMPALA-7078 - that may be enough to solve the problem. Look at CDH6.1, it does address a bunch of issues in this area more systematically - it moves the scan operations to use a much more robust memory throttling/reservation system (I spent a bunch of time last year working on problems in this general area). 1 GB might just not be enough to run a query with that many operators on the version of Impala that you're running. ... View more

any update on this? specifying columns from join when there is over 50 columns is error prune. ... View more

So it looks like column specific is only on a table without partitions (non-incremental) @hores that's incorrect, non-incremental compute stats works on partitioned tables and is generally the preferred method for collecting stats on partitioned tables.   We've generally tried to steer people away from incremental stats because of the size issues on large tables,   It would also be error-prone to use correctly and complex to implement - what happens if you compute incremental stats with different subsets of the columns? You can end up with different subsets of the columns on different partitions and then you have to somehow reconcile it all each time. ... View more

  @Zane- I'm late but can provide some additional insight.   I think the suggestion in the error message is a good one (I'm biased because I wrote it, but some thought went into it). "Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error". The general solution for this is to set up admission control with some memory limits so that memory doesn't get oversubscribed, and so that one query can't gobble up more memory than you like. I did a talk at strata that gave pointers on a lot of this things - https://conferences.oreilly.com/strata/strata-ca-2019/public/schedule/detail/73000   In this case you can actually see that query 2f4b5cff11212907:886aa1400000000 is using Total=78.60 GB memory, so that's likely your problem.   Impala's resource management is totally permissive out of the box and will happily let queries use up all the resources in the system like this.   I didn't see what version you're running, but there were a lot of improvements in this area (config options, OOM-avoidance, diagnostics) in CDH6.1+   There's various other angles you can take to improve this - if the queries using lots of memory are suboptimal, tuning them (maybe just computing stats) makes a big difference. You can also  ... View more

>"Your cluster does sound unhappy"   LOL.  I'd say more like pi$$ed.  🙂   > If it's a JVM issue, we've seen in some cases that increasing heap sizes helps. Setting ipc.client.rpc-timeout.ms to 60000 I'd say it's more like a set of overloaded namenodes.  And, according to the research I've done so far, for another problem, I have a theory that I can use a more up to date GC and it should increase the performance and reduce the number of Zookeeper failures we have.  Our ZK failures are sporadic and happen every 2 to 3 days.  Sometimes more sometimes less.  Moving ZK to separate nodes is not an option at this point and I'm doing all I can to try to reduce the number of failures short of moving the service. I'll check our settings on this and see if we can do one or both.  I suspect we have increased JVM heap already, but not sure?   >We've also seen the file handle cache that got enabled by default in CDH5.15 help a lot in reducing namenode load, I assume this is available before this version but was not enabled by default???  I'll look it up and see...   > I agree 100%. I think whoever named it was either overly optimistic and assumed there wouldn't be a significant gap in time, or it was named from the point of view of the code rather than the external system So, my question is, is there an indicator in the Query Details that indicates something was returned?  I know I get an initial set of results back.  Without that "fetch" metric meaning what the word actually says, I don't know what indicates how long it took to get the first set of records back???   Back to the original issue... Given that the issue appears to be the last query issued in Hue tends to show up as still executing 2.2 hours later and has already returned a count almost immediately.  Obviously, the parameters for idle timeouts for sessions and queries is not marking the session as closed.  Therefore appearing to still be executing: Is this causing resource issues because the session is being held open and appearing to be still executing?  I would assume so as it is waiting on a fetch of subsequent sets of records??? What parameter(s) will close the session from the last query executed?   Just to let you know, I've come in late to the game and am still learning CM and Cloudera Manager.  I understand a lot but with 1000s of parameters for all the apps and an ample supply of problems, it'll take a while.  🙂   Thanks for all your help.  It is nice to have a response on this forum.  The last couple posts were not very useful.  We do have a service contract and although I am one of the 3 admins, they are working on adding me to the support contract so I can put in tickets and get support directly. Until then, I appreciate the help! ... View more

I filed https://issues.apache.org/jira/browse/IMPALA-8807 to fix the docs. ... View more

Like @EricL said, this would be caused by some process updating files in the table in the background without a refresh in Impala. E.g. if you have a job that writes files directly into the table and can either write incomplete files or has Impala see the files before they are completely written (preferably you write the files in a temporary directory then move them into the table directory). Some usage patterns for hive might cause issues, e.g. INSERT OVERWRITE.   There was a related issue in Impala that could occur if you did an "INSERT OVERWRITE" from hive without a refresh from Impala: https://issues.apache.org/jira/browse/IMPALA-8561. Generally that workflow (insert overwrite without refresh) is problematic, but the symptoms were made more confusing by IMPALA-8561. ... View more

Probably, the version we are using is 5.12. ... View more

If I had to guess, the CDH installation is somehow broken and missing jar files. Impala depends on antlr so won't be able to run if that isn't present. The JARs should be part of the CDH parcel, e.g. in /opt/cloudera/parcels/CDH-<version>/lib/impala/lib ... View more