>"Your cluster does sound unhappy" LOL. I'd say more like pi$$ed. 🙂 > If it's a JVM issue, we've seen in some cases that increasing heap sizes helps. Setting ipc.client.rpc-timeout.ms to 60000 I'd say it's more like a set of overloaded namenodes. And, according to the research I've done so far, for another problem, I have a theory that I can use a more up to date GC and it should increase the performance and reduce the number of Zookeeper failures we have. Our ZK failures are sporadic and happen every 2 to 3 days. Sometimes more sometimes less. Moving ZK to separate nodes is not an option at this point and I'm doing all I can to try to reduce the number of failures short of moving the service. I'll check our settings on this and see if we can do one or both. I suspect we have increased JVM heap already, but not sure? >We've also seen the file handle cache that got enabled by default in CDH5.15 help a lot in reducing namenode load, I assume this is available before this version but was not enabled by default??? I'll look it up and see... > I agree 100%. I think whoever named it was either overly optimistic and assumed there wouldn't be a significant gap in time, or it was named from the point of view of the code rather than the external system So, my question is, is there an indicator in the Query Details that indicates something was returned? I know I get an initial set of results back. Without that "fetch" metric meaning what the word actually says, I don't know what indicates how long it took to get the first set of records back??? Back to the original issue... Given that the issue appears to be the last query issued in Hue tends to show up as still executing 2.2 hours later and has already returned a count almost immediately. Obviously, the parameters for idle timeouts for sessions and queries is not marking the session as closed. Therefore appearing to still be executing: Is this causing resource issues because the session is being held open and appearing to be still executing? I would assume so as it is waiting on a fetch of subsequent sets of records??? What parameter(s) will close the session from the last query executed? Just to let you know, I've come in late to the game and am still learning CM and Cloudera Manager. I understand a lot but with 1000s of parameters for all the apps and an ample supply of problems, it'll take a while. 🙂 Thanks for all your help. It is nice to have a response on this forum. The last couple posts were not very useful. We do have a service contract and although I am one of the 3 admins, they are working on adding me to the support contract so I can put in tickets and get support directly. Until then, I appreciate the help!
... View more
