Cloudera Labs
Provide feedback on Cloudera Labs
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can't start Kafka Service broker

Solved Go to solution
Highlighted

Can't start Kafka Service broker

Hi,

 

I successfully installed Kafka on my test cluster but when I followed same steps on production cluster I am running into an issue. I followed Cloudera documentation step by step. I was able to download, distributed and activated the Kafka Parcel. When I tried adding "Kafka-Cloudera-Labs" on any hadoop nodes and I was NOT able to get the broker up and running. Here is the error log that can be available.

 

2014-12-02 09:00:37,815 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected)

2014-12-02 09:00:37,853 INFO kafka.log.LogManager: Loading log '.gnome2'

2014-12-02 09:00:37,855 FATAL kafka.server.KafkaServerStartable: Fatal error during KafkaServerStable startup. Prepare to shutdown

java.lang.StringIndexOutOfBoundsException: String index out of range: -1

        at java.lang.String.substring(String.java:1911)

        at kafka.log.Log$.parseTopicPartitionName(Log.scala:740)

        at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogMa

        at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogMa

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimize

        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)

        at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)

        at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimize

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.log.LogManager.loadLogs(LogManager.scala:105)

        at kafka.log.LogManager.<init>(LogManager.scala:57)

        at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:293)

        at kafka.server.KafkaServer.startup(KafkaServer.scala:73)

        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:

        at kafka.Kafka$.main(Kafka.scala:46)

        at kafka.Kafka.main(Kafka.scala)

2014-12-02 09:00:37,858 INFO kafka.server.KafkaServer: [Kafka Server 159], shutt

 

I did exact same thing on a test cluster and it worked very well. Both production and test clusters are running same CM5.2 and CDH5.2.0 versions.

 

What could be potential issue and how to fix this issue?

 

Please help!!!

 

Thanks,

Silaphet

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Can't start Kafka Service broker

Contributor
Big thank you for being a Cloudera Labs user!

Yeah, /var/lib/kafka is the kafka logs (basically kafka data). /var/log/kafka is the actual error logs for the various components.

View solution in original post

6 REPLIES 6
Highlighted

Re: Can't start Kafka Service broker

Contributor
Check under /var/lib/kafka, there will be a file called .gnome2. Delete this and other non-kafka files (which is probably all files at this stage).
Highlighted

Re: Can't start Kafka Service broker

Hi,

 

There is no file called .gnome2 under /var/lib/kafka. There is only recovery-point-offset-checkpoint under this directory. However, on a working test cluster, there are two files under this directory. Here they are:

 

recovery-point-offset-checkpoint

replication-offset-checkpoint

 

The difference between these two clusters is test cluster has only one zookeeper compared to three zookeeper nodes on production. I'm not convinced that this would be root cause of it.

 

Do you know where else I should look at?

 

Thanks,

Silaphet

 

Re: Can't start Kafka Service broker

Contributor

Can you run ls -la /var/lib/kafka and post the results here?

Also look at Cloudera Manager for the Kafka log dir configuration and let us know what it is.

 

In general, /var/lib/kafka is the default log directory for Kafka. 

Kafka will assume that any subdirectory found there is a Kafka topic. If it isn't, you'll get the error you posted.

In this case, it found a directory called .gnome2.

 

See also: https://issues.apache.org/jira/browse/KAFKA-742

Highlighted

Re: Can't start Kafka Service broker

Hi,

 

I see they are hidden directories. After deleting both .gnome2 and .mozzila and I was able to get the broker up and running.

 

From CM, I can see /var/lib/kafka is the log directory. I guess /var/log/kafka is server log, is that correct?

 

Big thanks to you for a quick response.

 

Thanks,

Silaphet

Highlighted

Re: Can't start Kafka Service broker

Contributor
Big thank you for being a Cloudera Labs user!

Yeah, /var/lib/kafka is the kafka logs (basically kafka data). /var/log/kafka is the actual error logs for the various components.

View solution in original post

Highlighted

Re: Can't start Kafka Service broker

Contributor

Also need to check if Heap size value is updated.

Don't have an account?
Coming from Hortonworks? Activate your account here