question Re: Spark Kafa createDirectStream failed while createStream successful in Archives of Support Questions (Read Only)

Spark Kafa createDirectStream failed while createStream successful

zhusiyao — Thu, 12 May 2016 04:03:41 GMT

I'm testing spark streaming from Kafka code from Spark example in HDP 2.4 and Spark 1.6.

I tried KafkaWordCount.scala example successful with

`$ bin/run-example org.apache.spark.examples.streaming.KafkaWordCount zoo01,zoo02,zoo03 my-consumer-group topic1,topic2 1`

But DirectKafkaWordCount.scala failed with below query and error

bin/run-example streaming.DirectKafkaWordCount broker1-host:port,broker2-host:port topic1,topic2

16/05/11 13:52:02 INFO SimpleConsumer: Reconnect due to socket error: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
org.apache.spark.SparkException: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
        at org.apache.spark.streaming.kafka.KafkaCluster$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)
        at org.apache.spark.streaming.kafka.KafkaCluster$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)

Re: Spark Kafa createDirectStream failed while createStream successful

bleonhardi — Thu, 12 May 2016 15:00:43 GMT

This can have different reasons unfortunately I have seen the same error when using wrong hostnames ( not fully qualified, localhost instead of the full hostname ... ) I have also seen it in the context of security, i.e. when connecting to a kerberized kafka and providing a wrong jaas configuration, ...

First thing I would do is check the broker-host variable. Fully qualify it and see if that fixes anything.

Re: Spark Kafa createDirectStream failed while createStream successful

zhusiyao — Fri, 13 May 2016 01:42:02 GMT

I'm using exactly the same broker list and topics in both command. So I think that won't be the cause.

Re: Spark Kafa createDirectStream failed while createStream successful

zhusiyao — Fri, 13 May 2016 05:01:00 GMT

It's a known issue for HDP 2.4 release, BUG-51292. And there is no workaround....