Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark : Receiver Blocks Aggregated Block Metrics by Executor

Highlighted

Spark : Receiver Blocks Aggregated Block Metrics by Executor

Explorer

We have a spark streaming application receiving data from kafka... we see the application is slow and trying to find the reason for it....i see the Receiver Blocks Aggregated Block Metrics by Executor

Receiver Blocks  Aggregated Block Metrics by Executor

Was wondering if this distribution on only one executor has anything to do with it ? Any help will be greatly appreciated.

1 REPLY 1

Re: Spark : Receiver Blocks Aggregated Block Metrics by Executor

Expert Contributor

You only have one reciever, so that executor will contain all the blocks.  To distribute the blocks to other executors, you have a few options:

 

1.  Call repartition on your RDD

2.  Increase replication and the same block will be on mutliple executors.

3.  Create multiple receivers and union RDD later.

Don't have an account?
Coming from Hortonworks? Activate your account here