Recently upgraded Cloudera from 5.4 to 5.12 but running 5.11 components
so Impala 2.8.0+cdh5.11.1+0
Also recently moved Hive Metastore database from postgres to MySQL
Currently I have found that Impala is reacting slowly to executing queiries and executing them at all
I have a 7 node cluster and only 5 real users using Impala
In searching the logs
I am getting the following error
E0810 18:01:54.606694 25271 ShortCircuitCache.java:215] ShortCircuitCache(0xba3834e): failed to release short-circuit shared memory slot Slot(slotIdx=17, shm=DfsClientShm(5db0b20a1ec13fd9af815db2d22cd894)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs-sockets/dn. Closing shared memory segment.
Java exception follows:
java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 5db0b20a1ec13fd9af815db2d22cd894
at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I also see the following as well
cc:278
channel send status:
Sender timed out waiting for receiver fragment instance: 3d4e8ea794a486e7:a74d612000000010
