Impala running slow after upgrade

scratch28 — Fri, 16 Sep 2022 12:04:35 GMT

Recently upgraded Cloudera from 5.4 to 5.12 but running 5.11 components

so Impala 2.8.0+cdh5.11.1+0

Also recently moved Hive Metastore database from postgres to MySQL

Currently I have found that Impala is reacting slowly to executing queiries and executing them at all

I have a 7 node cluster and only 5 real users using Impala

In searching the logs

I am getting the following error

E0810 18:01:54.606694 25271 ShortCircuitCache.java:215] ShortCircuitCache(0xba3834e): failed to release short-circuit shared memory slot Slot(slotIdx=17, shm=DfsClientShm(5db0b20a1ec13fd9af815db2d22cd894)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs-sockets/dn.  Closing shared memory segment.
Java exception follows:
java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 5db0b20a1ec13fd9af815db2d22cd894
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I also see the following as well

cc:278	
channel send status: 
Sender timed out waiting for receiver fragment instance: 3d4e8ea794a486e7:a74d612000000010

Re: Impala running slow after upgrade

Tim Armstrong — Mon, 14 Aug 2017 18:49:57 GMT

Do you have a profile for any of the slow queries?

I see some HDFS bugs that suggest that there may be problems along these lines when running mismatched versions of the HDFS client and HDFS datanode: https://issues.apache.org/jira/browse/HDFS-8070.

Re: Impala running slow after upgrade

scratch28 — Wed, 16 Aug 2017 22:55:31 GMT

Agent and edge node where HUE was running was not upgraded

question Impala running slow after upgrade in Archives of Support Questions (Read Only)

Impala running slow after upgrade

Re: Impala running slow after upgrade

Re: Impala running slow after upgrade