Support Questions

Find answers, ask questions, and share your expertise

Error while running structured streaming example “WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected”

avatar

While running "structured_kafka_wordcount.py" example given in "https://github.com/apache/spark/blob/v2.3.1/examples/src/main/python/sql/streaming/structured_kafka_..." I got following error: "WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected"

I was able to read the content of a topic from kafka as given it the example https://github.com/apache/spark/blob/master/examples/src/main/python/streaming/kafka_wordcount.py

I submitted these job with command "bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1 examples/src/main/python/sql/streaming/structured_kafka_wordcount.py ip-10-28-3-35.ec2.internal:2181 subscribe fifa2"

<code>[root@centos spark2]# bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1 examples/src/main/python/sql/streaming/structured_kafka_wordcount.py ip-10-28-3-35.ec2.internal:2181 subscribe fifa2
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/usr/hdp/2.6.5.0-292/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.spark#spark-sql-kafka-0-10_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
        confs: [default]
        found org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1 in central
        found org.apache.kafka#kafka-clients;0.10.0.1 in central
        found net.jpountz.lz4#lz4;1.3.0 in central
        found org.xerial.snappy#snappy-java;1.1.2.6 in central
        found org.slf4j#slf4j-api;1.7.16 in central
        found org.spark-project.spark#unused;1.0.0 in central
:: resolution report :: resolve 642ms :: artifacts dl 15ms
        :: modules in use:
        net.jpountz.lz4#lz4;1.3.0 from central in [default]
        org.apache.kafka#kafka-clients;0.10.0.1 from central in [default]
        org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1 from central in [default]
        org.slf4j#slf4j-api;1.7.16 from central in [default]
        org.spark-project.spark#unused;1.0.0 from central in [default]
        org.xerial.snappy#snappy-java;1.1.2.6 from central in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   6   |   0   |   0   |   0   ||   6   |   0   |
        ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
        confs: [default]
        0 artifacts copied, 6 already retrieved (0kB/17ms)
18/06/22 06:57:26 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
18/06/22 06:57:26 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
^Z
[11]+  Stopped                 bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1 examples/src/main/python/sql/streaming/structured_kafka_wordcount.py ip-10-28-3-35.ec2.internal:2181 subscribe fifa2
1 ACCEPTED SOLUTION

avatar
Expert Contributor

@Mahesh

Hi,

Could you please give Kafka broker hostname:port instead of zookeeper_host:2181 in the command? The following error:

WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected

Means that its unable to reach to correct Kafka broker because zookeeper information has been given. Please let me know how it goes.

Thanks!

View solution in original post

4 REPLIES 4

avatar
Expert Contributor

@Mahesh

Hi,

Could you please give Kafka broker hostname:port instead of zookeeper_host:2181 in the command? The following error:

WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected

Means that its unable to reach to correct Kafka broker because zookeeper information has been given. Please let me know how it goes.

Thanks!

avatar

Hi @dbains,

I Checked the kafka broker port on my HDP cluster and it was 6667. Although I was confused between Bootstrap Server and Kafka Broker, I was giving zookeeper_ip:2181. It worked after passing ip-address:6667.

avatar
Expert Contributor

@Mahesh

Glad it worked!

Thanks!

avatar
Expert Contributor

@Mahesh

Hi, Did the suggestion help?

Thanks!