17/10/31 06:53:41 INFO Executor: Finished task 44.0 in stage 121.0 (TID 11558). 2938 bytes result sent to driver
17/10/31 08:08:46 INFO CoarseGrainedExecutorBackend: Got assigned task 11771
Every executor in application is waiting for atleast an hour between this lines, when enabled debug logs we see org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$CacheCleaner is happening in middle.
Any insight how to reduce this time would be helpful.