Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark process failing to receive data from the Kafka queue in yarn-client mode.

Highlighted

Spark process failing to receive data from the Kafka queue in yarn-client mode.

I am trying to run following code using yarn-client mode in but getting slow readprocessor error mentioned below but the code works just fine in the local mode.  Any pointer is really appreciated.

 

Line of code to receive data from the Kafka Queue:

JavaPairReceiverInputDStream<String, String> messages =  KafkaUtils.createStream(jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, kafkaTopicMap, StorageLevel.MEMORY_ONLY());

 

JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {

      public String call(Tuple2<String, String> tuple2) {

                              LOG.info(" &&&&&&&&&&&&&&&&&&&& Input json stream data  " +  tuple2._2);

        return tuple2._2();

      }

    });

 

 

Error Details:

016-02-05 11:44:00 WARN DFSClient:975 - Slow ReadProcessor read fields took 30
011ms (threshold=30000ms); ack: seqno: 1960 reply: 0 reply: 0 reply: 0 downstrea
mAckTimeNanos: 1227280, targets: [DatanodeInfoWithStorage[10.0.0.245:50010,DS-a5
5d9212-3771-4936-bbe7-02035e7de148,DISK], DatanodeInfoWithStorage[10.0.0.243:500
10,DS-231b9915-c2e2-4392-b075-8a52ba1820ac,DISK], DatanodeInfoWithStorage[10.0.0
.244:50010,DS-6b8b5814-7dd7-4315-847c-b73bd375af0e,DISK]]
2016-02-05 11:44:00 INFO BlockManager:59 - Removing RDD 1954
2016-02-05 11:44:00 INFO MapPartitionsRDD:59 - Removing RDD 1955 from persisten

Don't have an account?
Coming from Hortonworks? Activate your account here