Support Questions
Find answers, ask questions, and share your expertise

flume error with "Failed to find leader".

Highlighted

flume error with "Failed to find leader".

Hi - i'm trying to do setup "kafka(as source) to flume to hdfs" but getting below error message in flume logs.

[flume_tstr400367-1483611425172-236ed292-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1483611425172-236ed292-leader-finder-thread], Failed to find leader for Set([connect-test,0])

java.lang.NullPointerException

at org.apache.kafka.common.utils.Utils.formatAddress(Utils.java:312)

at kafka.cluster.Broker.connectionString(Broker.scala:62)

at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89)

at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)

at scala.collection.AbstractTraversable.map(Traversable.scala:105)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:89)

at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

can you please help in fixing the issue

10 REPLIES 10
Highlighted

Re: flume error with "Failed to find leader".

Master Collaborator

Looks like the issue may be with Kafka brokers, can you check if the Kafka services are running fine?

Highlighted

Re: flume error with "Failed to find leader".

Super Guru
@sathish jeganathan

Can you please check if you have correct kafka jars in flume's lib directory?

Please have a look at - https://issues.apache.org/jira/browse/FLUME-2790

Highlighted

Re: flume error with "Failed to find leader".

i just copied all the jars from /kafka-server/libs to /flume-server/lib... but still am seeing the same error..

kafka version - 0.9

flume version - Flume 1.5.2.2.4.2.0-258

thanks,

sathish

Re: flume error with "Failed to find leader".

i'm not able to start the agent from ambari. i've started ambari service from amabri but not ambari agent. its neither logging anything nor starting... all i can see is "no flume to display".

below are my config details....

kafka source and hdfs sink

#flume agent config

agent.sources = kafka-source-1

agent.channels = hdfs-channel-1

agent.sinks = hdfs-sink-1

#source properties

agent.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource

agent.sources.kafka-source-1.zookeeperConnect = tstr400360.xyz:2181

agent.sources.kafka-source-1.topic = connect-test

agent.sources.kafka-source-1.batchSize = 100

agent.sources.kafka-source-1.channels = hdfs-channel-1

#sink properties

agent.sinks.hdfs-sink-1.channel = hdfs-channel-1

agent.sinks.hdfs-sink-1.type = hdfs

agent.sinks.hdfs-sink-1.hdfs.writeFormat = Text

agent.sinks.hdfs-sink-1.hdfs.fileType = DataStream

agent.sinks.hdfs-sink-1.hdfs.filePrefix = test-events

agent.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true

agent.sinks.hdfs-sink-1.hdfs.path = /user/flume/stage

agent.sinks.hdfs-sink-1.hdfs.rollCount=100

agent.sinks.hdfs-sink-1.hdfs.rollSize=0

# channel properties

agent.channels.hdfs-channel-1.type = memory

agent.channels.hdfs-channel-1.capacity = 10000

agent.channels.hdfs-channel-1.transactionCapacity = 1000

thanks,

sathish

Highlighted

Re: flume error with "Failed to find leader".

i was able to successfully create flume agent with normal spooldir source... its failing or not even starting the agent for kafka soruce

Highlighted

Re: flume error with "Failed to find leader".

Explorer

Stack trace seems to suggest ... some problem with your zookeeperConnect setting.

Highlighted

Re: flume error with "Failed to find leader".

i brought up the agent with kafka as source.. however hdfs sink is not working ... and am seeing below error in agent log.

WARN [flume_tstr400367-1484117904365-8f70ac48-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1484117904365-8f70ac48-leader-finder-thread], Failed to find leader for Set([connect-test,0])

java.lang.NullPointerException

at org.apache.kafka.common.utils.Utils.formatAddress(Utils.java:312)

at kafka.cluster.Broker.connectionString(Broker.scala:62)

at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89)

at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)

at scala.collection.AbstractTraversable.map(Traversable.scala:105)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:89)

at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

Highlighted

Re: flume error with "Failed to find leader".

i just wanna doubel check the compatibility between all these three things..

flume version :

Flume 1.5.2.2.4.2.0-258

Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Revision: e2cd47aa3cc959a659a97dfe800138d11744fa62

Compiled by jenkins on Mon Apr 25 08:49:00 UTC 2016

From source with checksum 2cd1bf99fd2acccc72128b0254e445ea

kafka version : 0.9.0.2.4.2.0-258

security protocol : plaintextsasl

thanks,

sathish

Highlighted

Re: flume error with "Failed to find leader".

please someone help me with below error

[flume_tstr400367-1484119363347-a597ba80-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1484119363347-a597ba80-leader-finder-thread], Failed to find leader for Set([connect-test,0])

java.lang.NullPointerException

thanks,

sathish