Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Impala running slow after upgrade

avatar
Contributor

Recently upgraded Cloudera from 5.4 to 5.12 but running 5.11 components

so Impala 2.8.0+cdh5.11.1+0

 

Also recently moved Hive Metastore database from postgres to MySQL

Currently I have found that Impala is reacting slowly to executing queiries and executing them at all

I have a 7 node cluster and only 5 real users using Impala 

In searching the logs 

I am getting the following error 

 

E0810 18:01:54.606694 25271 ShortCircuitCache.java:215] ShortCircuitCache(0xba3834e): failed to release short-circuit shared memory slot Slot(slotIdx=17, shm=DfsClientShm(5db0b20a1ec13fd9af815db2d22cd894)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs-sockets/dn.  Closing shared memory segment.
Java exception follows:
java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 5db0b20a1ec13fd9af815db2d22cd894
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I also see the following as well

cc:278	
channel send status: 
Sender timed out waiting for receiver fragment instance: 3d4e8ea794a486e7:a74d612000000010
1 ACCEPTED SOLUTION

avatar

Do you have a profile for any of the slow queries?


I see some HDFS bugs that suggest that there may be problems along these lines when running mismatched versions of the HDFS client and HDFS datanode: https://issues.apache.org/jira/browse/HDFS-8070.

View solution in original post

2 REPLIES 2

avatar

Do you have a profile for any of the slow queries?


I see some HDFS bugs that suggest that there may be problems along these lines when running mismatched versions of the HDFS client and HDFS datanode: https://issues.apache.org/jira/browse/HDFS-8070.

avatar
Contributor

Agent and edge node where HUE was running was not upgraded