flume error with "Failed to find leader".

Sathish Jeganathan · ‎01-05-2017

Hi - i'm trying to do setup "kafka(as source) to flume to hdfs" but getting below error message in flume logs.

[flume_tstr400367-1483611425172-236ed292-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1483611425172-236ed292-leader-finder-thread], Failed to find leader for Set([connect-test,0])

java.lang.NullPointerException

at org.apache.kafka.common.utils.Utils.formatAddress(Utils.java:312)

at kafka.cluster.Broker.connectionString(Broker.scala:62)

at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)

at scala.collection.AbstractTraversable.map(Traversable.scala:105)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:89)

at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

can you please help in fixing the issue

deepesh1 · ‎01-05-2017

Looks like the issue may be with Kafka brokers, can you check if the Kafka services are running fine?

KuldeepK · ‎01-05-2017

@sathish jeganathan

Can you please check if you have correct kafka jars in flume's lib directory?

Please have a look at - https://issues.apache.org/jira/browse/FLUME-2790

Sathish Jeganathan · ‎01-06-2017

i just copied all the jars from /kafka-server/libs to /flume-server/lib... but still am seeing the same error..

kafka version - 0.9

flume version - Flume 1.5.2.2.4.2.0-258

thanks,

sathish

Sathish Jeganathan · ‎01-10-2017

i'm not able to start the agent from ambari. i've started ambari service from amabri but not ambari agent. its neither logging anything nor starting... all i can see is "no flume to display".

below are my config details....

kafka source and hdfs sink

#flume agent config

agent.sources = kafka-source-1

agent.channels = hdfs-channel-1

agent.sinks = hdfs-sink-1

#source properties

agent.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource

agent.sources.kafka-source-1.zookeeperConnect = tstr400360.xyz:2181

agent.sources.kafka-source-1.topic = connect-test

agent.sources.kafka-source-1.batchSize = 100

agent.sources.kafka-source-1.channels = hdfs-channel-1

#sink properties

agent.sinks.hdfs-sink-1.channel = hdfs-channel-1

agent.sinks.hdfs-sink-1.type = hdfs

agent.sinks.hdfs-sink-1.hdfs.writeFormat = Text

agent.sinks.hdfs-sink-1.hdfs.fileType = DataStream

agent.sinks.hdfs-sink-1.hdfs.filePrefix = test-events

agent.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true

agent.sinks.hdfs-sink-1.hdfs.path = /user/flume/stage

agent.sinks.hdfs-sink-1.hdfs.rollCount=100

agent.sinks.hdfs-sink-1.hdfs.rollSize=0

# channel properties

agent.channels.hdfs-channel-1.type = memory

agent.channels.hdfs-channel-1.capacity = 10000

agent.channels.hdfs-channel-1.transactionCapacity = 1000

thanks,

sathish

Sathish Jeganathan · ‎01-10-2017

i was able to successfully create flume agent with normal spooldir source... its failing or not even starting the agent for kafka soruce

roshan1 · ‎01-10-2017

Stack trace seems to suggest ... some problem with your zookeeperConnect setting.

Sathish Jeganathan · ‎01-11-2017

i brought up the agent with kafka as source.. however hdfs sink is not working ... and am seeing below error in agent log.

WARN [flume_tstr400367-1484117904365-8f70ac48-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1484117904365-8f70ac48-leader-finder-thread], Failed to find leader for Set([connect-test,0])