Support Questions
Find answers, ask questions, and share your expertise
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

Finding Maximum value from the Spark Streaming data and sending the result to the other topic in Kafka using Spark-Kafka Approach.

Hi All,

Can someone please help be with the below requirement.

I have 2 topics called feed1 and feed2 in kafka where in feed1 produces the data and feed2 receives the data.

I have written a spark-kafka code wherein consumes data from feed1 and save into HDFS.

Now the requirement is to find out maximum value from the streaming Kafka-spark code and send back the resultant value to the other feed which is feed2.

Ex: feed1 (Produces data)

Spark-Kafka Application --> Consumes data from Feed1 ----> HDFS

Also Find the Max ---> in the same application using 1 min as a window and --------------> Send that max value to the Feed2

Can some one help me whether this can be achieved or not. If yes, please give me some idea or pseudo code.

Thanks in advance


Vijay Kumar