Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.6: Spark 2.1 Streaming not receiving data in the provided batch interval

Highlighted

HDP 2.6: Spark 2.1 Streaming not receiving data in the provided batch interval

New Contributor

I have created Spark Streaming application which receives data from Kafka topic.

Batch interval : 5 Seconds

Messages available in the topic: 10

Processing logic takes: 1 min

Finally the output converted into JSON and Published into another Kafka Topic.

Problem 1:

Final output not publishing into another Kafka topic.

def publishIntoKafka(predictedDataRDD: RDD[String], kafkaBroker: String): Unit = {
logger.info(s"Publishing into Kafka................") 
// SEND TO KAFKA 
predictedDataRDD.foreachPartition(partition => 
{
// Print statements in this section are shown in the executor's stdout logs
val kafkaOpTopic = "anomalyDetectedTest" 
val props = new HashMap[String, Object]() 
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBroker) props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer") 
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer") val producer = new KafkaProducer[String, String](props) 
partition.foreach(record => { 
val data = record.toString // As as debugging technique, users can write to DBFS to verify that records are being written out val message = new ProducerRecord[String, String](kafkaOpTopic, null, data) producer.send(message) }) producer.close() }) }

Problem 2:

If I publish the message again after certain time period lets say after 10 seconds then that message does not consumed by Spark Streaming. The Spark UI Jobs and Stages remains same and nothing happening.

I have make sure the Spark Streaming application is running in YARN in the Resource Manager UI.

I did not found any errors in the YARN application log,

yarn logs -applicationId application_1499166717658_0003

Help me if any one knows the solution and let me know if there is any mistakes in my side.

Thanks,

Uvaraj.S

Don't have an account?
Coming from Hortonworks? Activate your account here