question Re: HOW TO INTEGRATE OR CONSUME DATA FROM KAFKA TO SPARK STREAM in Archives of Support Questions (Read Only)

HOW TO INTEGRATE OR CONSUME DATA FROM KAFKA TO SPARK STREAM

mail_sarfarazkh — Fri, 29 Apr 2016 16:33:09 GMT

Can somebody tell me the data comeing from twitter to kafka succesfull now i want to process further in spark and store into hdfs please provide the java code for the same or any link that i can refer to

Re: HOW TO INTEGRATE OR CONSUME DATA FROM KAFKA TO SPARK STREAM

pvillard — Fri, 29 Apr 2016 16:37:19 GMT

Hi,

You will find an article here : https://community.hortonworks.com/articles/25726/spark-streaming-explained-kafka-to-phoenix.html

Writing to HDFS can be easily achieved with RDD.saveAsTextFile() for example.

HTH

Re: HOW TO INTEGRATE OR CONSUME DATA FROM KAFKA TO SPARK STREAM

jyadav — Sat, 30 Apr 2016 02:55:12 GMT

@sarfarazkhan pathanI think you can't ignore this guy 🙂 Awesome explanation with code examples.

http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/

Re: HOW TO INTEGRATE OR CONSUME DATA FROM KAFKA TO SPARK STREAM

vvaks — Mon, 02 May 2016 23:46:03 GMT

@sarfarazkhan pathan

You will need the following in your Maven POM:

	<dependency>
		<groupId>org.apache.spark</groupId>
		<artifactId>spark-streaming-kafka_2.10</artifactId>
		<version>VERSION</version>
	</dependency>

Then create a stream as follows:

Map<String, Integer> kafkaTopics = new HashMap<String, Integer>();
kafkaTopics.put("TopicName", 1);
SparkConf sparkConf = new SparkConf();
	
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(batchSize));	
JavaPairReceiverInputDStream<String, String> kafkaStream = 
KafkaUtils.createStream(jssc, Constants.zkConnString,"spark-streaming-consumer-group", kafkaTopics);
//kafkaStream.print();
JavaPairDStream<String, String> deviceStream = kafkaStream;

Re: HOW TO INTEGRATE OR CONSUME DATA FROM KAFKA TO SPARK STREAM

Justin_Watkins — Wed, 04 May 2016 18:56:46 GMT

I agree with @Jitendra Yadav. The blog-posts of Michael Noll are excellent reading, especially in the realm of Kafka.