Posts: 120
Registered: ‎10-15-2014
Accepted Solution

Impala running slow after upgrade

Recently upgraded Cloudera from 5.4 to 5.12 but running 5.11 components

so Impala 2.8.0+cdh5.11.1+0


Also recently moved Hive Metastore database from postgres to MySQL

Currently I have found that Impala is reacting slowly to executing queiries and executing them at all

I have a 7 node cluster and only 5 real users using Impala 

In searching the logs 

I am getting the following error 


E0810 18:01:54.606694 25271] ShortCircuitCache(0xba3834e): failed to release short-circuit shared memory slot Slot(slotIdx=17, shm=DfsClientShm(5db0b20a1ec13fd9af815db2d22cd894)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs-sockets/dn.  Closing shared memory segment.
Java exception follows: ERROR_INVALID: there is no shared memory segment registered with shmId 5db0b20a1ec13fd9af815db2d22cd894
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$
	at java.util.concurrent.Executors$
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(
	at java.util.concurrent.ScheduledThreadPoolExecutor$
	at java.util.concurrent.ThreadPoolExecutor.runWorker(
	at java.util.concurrent.ThreadPoolExecutor$

I also see the following as well

channel send status: 
Sender timed out waiting for receiver fragment instance: 3d4e8ea794a486e7:a74d612000000010
Cloudera Employee
Posts: 302
Registered: ‎07-29-2015

Re: Impala running slow after upgrade

Do you have a profile for any of the slow queries?

I see some HDFS bugs that suggest that there may be problems along these lines when running mismatched versions of the HDFS client and HDFS datanode:

Posts: 120
Registered: ‎10-15-2014

Re: Impala running slow after upgrade

Agent and edge node where HUE was running was not upgraded