Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Java.io.IOException: Connection to kafka VIA port 6667 failed

Highlighted

Java.io.IOException: Connection to kafka VIA port 6667 failed

We have 3 kafka machines and 3 zookper machines in our hadoop cluster

kafka machines are separated from the zookeper machines , and all machines are with OS - redhat 7.x

under /var/log/kafka we can see the following , java.io.IOException that displayed in server.log


at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2018-10-07 07:42:13,342] WARN [ReplicaFetcherThread-0-1007], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@407ac4e4 (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to kafka03.hfge.com:6667 (id: 1007 rack: null) failed at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93) at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:248) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2018-10-07 07:42:13,551] WARN [ReplicaFetcherThread-0-1002], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@2f9f8ecd (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to kafka07.hfge.com:6667 (id: 1002 rack: null) failed at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93) at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:248) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2018-10-07 07:42:15,343] WARN [ReplicaFetcherThread-0-1007], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4dafd0d9 (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to kafka03.hfge.com:6667 (id: 1007 rack: null) failed at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93) at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:248) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238) at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2018-10-07 07:42:15,552] WARN [ReplicaFetcherThread-0-1002], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5ce51081 (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to kafka07.hfge.com:6667 (id: 1002 rack: null) failed



additional with this , we also have a problem that kafka disconnected from the zookeeper after some time , this means that the broker id deleted from the zookeeper ( from zoo session /brokers/ids )

    cd /usr/hdp/current/zookeeper-server/bin
    ./zkCli.sh
    ls /brokers/ids 


any idea how to find the cause of the - java.io.IOException: Connection to kafka03.hfge.com:6667

Michael-Bronson
Don't have an account?
Coming from Hortonworks? Activate your account here