I successfully installed Kafka on my test cluster but when I followed same steps on production cluster I am running into an issue. I followed Cloudera documentation step by step. I was able to download, distributed and activated the Kafka Parcel. When I tried adding "Kafka-Cloudera-Labs" on any hadoop nodes and I was NOT able to get the broker up and running. Here is the error log that can be available.
2014-12-02 09:00:37,815 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected)
2014-12-02 09:00:37,853 INFO kafka.log.LogManager: Loading log '.gnome2'
2014-12-02 09:00:37,855 FATAL kafka.server.KafkaServerStartable: Fatal error during KafkaServerStable startup. Prepare to shutdown
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
2014-12-02 09:00:37,858 INFO kafka.server.KafkaServer: [Kafka Server 159], shutt
I did exact same thing on a test cluster and it worked very well. Both production and test clusters are running same CM5.2 and CDH5.2.0 versions.
What could be potential issue and how to fix this issue?
There is no file called .gnome2 under /var/lib/kafka. There is only recovery-point-offset-checkpoint under this directory. However, on a working test cluster, there are two files under this directory. Here they are:
The difference between these two clusters is test cluster has only one zookeeper compared to three zookeeper nodes on production. I'm not convinced that this would be root cause of it.
Do you know where else I should look at?
Can you run ls -la /var/lib/kafka and post the results here?
Also look at Cloudera Manager for the Kafka log dir configuration and let us know what it is.
In general, /var/lib/kafka is the default log directory for Kafka.
Kafka will assume that any subdirectory found there is a Kafka topic. If it isn't, you'll get the error you posted.
In this case, it found a directory called .gnome2.
I see they are hidden directories. After deleting both .gnome2 and .mozzila and I was able to get the broker up and running.
From CM, I can see /var/lib/kafka is the log directory. I guess /var/log/kafka is server log, is that correct?
Big thanks to you for a quick response.