Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

not getting any data in kafka consumer using spark structure streaming

Highlighted

not getting any data in kafka consumer using spark structure streaming

New Contributor

I have created a topic and ingest some sample data using producer .

But I am not able to consume those using spark structure streaming using below streaming code but it works fine using only kafka consumer code.


/usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic


/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list hadoopmaster:6667 --topic test_topic

>abc

>xyz

>mno


spark-shell --packages "org.apache.spark:spark-streaming-kafka_2.11:1.6.3" --packages "org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2"

scala > val df1 = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "hadoopmaster:6667").option("subscribe", "test_topic").load()

scala > df1.writeStream.format("console").option("truncate",false).start().awaitTermination()


Here no data is coming.



Don't have an account?
Coming from Hortonworks? Activate your account here