Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Finding Maximum value from the Spark Streaming data and sending the result to the other topic in Kafka using Spark-Kafka Approach.

Highlighted

Finding Maximum value from the Spark Streaming data and sending the result to the other topic in Kafka using Spark-Kafka Approach.

Hi All,

Can someone please help be with the below requirement.

I have 2 topics called feed1 and feed2 in kafka where in feed1 produces the data and feed2 receives the data.

I have written a spark-kafka code wherein consumes data from feed1 and save into HDFS.

Now the requirement is to find out maximum value from the streaming Kafka-spark code and send back the resultant value to the other feed which is feed2.

Ex: feed1 (Produces data)

Spark-Kafka Application --> Consumes data from Feed1 ----> HDFS

Also Find the Max ---> in the same application using 1 min as a window and --------------> Send that max value to the Feed2

Can some one help me whether this can be achieved or not. If yes, please give me some idea or pseudo code.

Thanks in advance

Regards,

Vijay Kumar

Don't have an account?
Coming from Hortonworks? Activate your account here