Hi Community, We are using AWS Network Load Balancer to balance out the traffic between 6 impala daemons.  Recently we started facing issues where 2 Impala daemons won't receive query and just hangs the connection over port 21050.  On further investigation we found that there were around 5k approx CLOSE_WAIT statuses for the connection between the LB and Impala daemon.    tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:64135 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:58169 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:64652 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:62075 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:52393 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:41447 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:47034 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:49452 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:28327 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:52498 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:21168 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:40079 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:35664 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:4191 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:14935 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:63036 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:5158 CLOSE_WAIT 11084/impalad tcp 1 0 10.XXX.XXX.68:21050 10.XXX.XXX.80:60134 CLOSE_WAIT 11084/impalad   Everytime I restart the daemons, the CLOSE_WAIT disappears and the connection establishes successfully but after few minutes these CLOSE_WAIT statuses piles up and chokes the connection again. Our cluster is Kerberized and TLS-SSL enabled. The NLB is internal and there is only one user using it through JDBC driver.   I'm stuck with this issue for over a week now, any suggestion will be very much helpful.   Thank you. ... View more

Hi Community, We are using Cloudera Altus Director 6.0.1 for cluster deployments.  Every now and then the WebUI stops working even though the server is up and running. When I check the logs, for most repeated occurrence is: java.lang.OutOfMemoryError: GC overhead limit exceeded [2019-09-23 13:08:23.894 +0000] WARN [qtp1613332278-15-acceptor-0@6caeba36-ServerConnector@1f9d6c7b{HTTP/1.1,[http/1.1]}{0.0.0.0:7189}] - - - - - o.e.jetty.server.AbstractConnector: java.lang.OutOfMemoryError: GC overhead limit exceeded [2019-09-23 13:08:50.803 +0000] DEBUG [qtp1613332278-15-acceptor-0@6caeba36-ServerConnector@1f9d6c7b{HTTP/1.1,[http/1.1]}{0.0.0.0:7189}] - - - - - org.eclipse.jetty.io.ManagedSelector: Queued change org.eclipse.jetty.io.ManagedSelector$Accept@12eec13c on org.eclipse.jetty.io.ManagedSelector@1aed6f0b id=0 keys=-1 selected=-1 updates=9 [2019-09-23 13:08:56.437 +0000] WARN [qtp1613332278-15] - - - - - o.e.j.util.thread.QueuedThreadPool: Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@1a7a9892 in QueuedThreadPool[qtp1613332278]@60297f36{STARTED,8<=8<=200,i=5,q=0}[ReservedThreadExecutor@6a078481{s=1/4,p=0}] When I restart the server, it fixes the issue.  I want to know if anyone else if also facing this same issue. And if there is any fix for that.   Thank you. ... View more

Hi Community, I'm working on adding load balancer in front of my Impala deamons.  I will be using AWS ELB As this is my first time I have few queries that are bothering me: When I create ELB which SSL certificate should I provide, the root CA certificate or the ones that cloudera manager creates while configuring AUTO_TLS, for example cm-auto-global_cacerts.pem? (My Cluster is already kerberized and TLS/SSL is enabled) Once the ELB is created, I will have to add the information to Impala service setting in CM and regenerate the Kerberos Credential. Will I be able to revert if something goes wrong in the process? I mean, if I remove the added configuration from Impala service setting in CM and restart, will it safely go back to working as before?   Thank you in advance. ... View more

Hello Community, I have a CDH 5.14 cluster hosted on EC2 machines which is kerberized with Active Directory and have Sentry for authorization of databases.  I want to use SSSD to secure my Linux hosts (RHEL 7.x) with Active Directory.  I have been going through this post but there are few queries which are bothering me to proceed forwards: 1. There are service-users (Hive, YARN, etc.) in AD that are already created during Kerberization of my cluster. So, if I go ahead and implement SSSD, then will these pre-existing service-users be able to communicate? 2. If something goes wrong will I be able to rollback? If yes, how?   ... View more

Scenario: I submit a heavy query in Impala through Hue which is expected to take around 2-3 hours. So, if my Hue session times out due to inactivity then will my query be still running in the background or will it get cancelled?    Hue's idle session timeout has been set to 20 mins and rest all other configurations are left to default values. ... View more