Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Executor Waiting

Spark Executor Waiting

Cloudera Employee
17/10/31 06:53:41 INFO Executor: Finished task 44.0 in stage 121.0 (TID 11558). 2938 bytes result sent to driver
17/10/31 08:08:46 INFO CoarseGrainedExecutorBackend: Got assigned task 11771

Every executor in application is waiting for atleast an hour between this lines, when enabled debug logs we see org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$CacheCleaner is happening in middle.

Any insight how to reduce this time would be helpful.

Don't have an account?
Coming from Hortonworks? Activate your account here