Reply
Contributor
Posts: 118
Registered: ‎10-15-2014
Accepted Solution

Impala running slow after upgrade

Recently upgraded Cloudera from 5.4 to 5.12 but running 5.11 components

so Impala 2.8.0+cdh5.11.1+0

 

Also recently moved Hive Metastore database from postgres to MySQL

Currently I have found that Impala is reacting slowly to executing queiries and executing them at all

I have a 7 node cluster and only 5 real users using Impala 

In searching the logs 

I am getting the following error 

 

E0810 18:01:54.606694 25271 ShortCircuitCache.java:215] ShortCircuitCache(0xba3834e): failed to release short-circuit shared memory slot Slot(slotIdx=17, shm=DfsClientShm(5db0b20a1ec13fd9af815db2d22cd894)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs-sockets/dn.  Closing shared memory segment.
Java exception follows:
java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 5db0b20a1ec13fd9af815db2d22cd894
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I also see the following as well

cc:278	
channel send status: 
Sender timed out waiting for receiver fragment instance: 3d4e8ea794a486e7:a74d612000000010
Cloudera Employee
Posts: 275
Registered: ‎07-29-2015

Re: Impala running slow after upgrade

Do you have a profile for any of the slow queries?


I see some HDFS bugs that suggest that there may be problems along these lines when running mismatched versions of the HDFS client and HDFS datanode: https://issues.apache.org/jira/browse/HDFS-8070.

Contributor
Posts: 118
Registered: ‎10-15-2014

Re: Impala running slow after upgrade

Agent and edge node where HUE was running was not upgraded 

Announcements