Reply
Explorer
Posts: 18
Registered: ‎05-02-2014
Accepted Solution

Kafka log directory warning

We are evaluating Kafka installation in ou  CHD5.3 cluster.

Due to our directory structure we had to move the log.dir to a non-default location (default is /var/log/kafka)

We changed value of log.dir and all dependant values in log4j.properties.

After this change, Cloudera Manager still shows the status of all brokers as bad, because the free space under default log directory is not sufficient. So Monitoring does not seem to read the value of log.dir from log4j-properties of kafka_broker.

Is there a way how to change Kafka log directory so that it is accepted by Monitoring automatically?

Cloudera Employee
Posts: 31
Registered: ‎10-16-2013

Re: Kafka log directory warning

I'm glad you got the test cluster working :)

 

For the log directory, Kafka has two types of logs:

- Transaction log - these contain the actual data written to Kafka, so they are critical to the normal operation of the product. We store those in log.dir which defaults to /var/local/kafka/data

 

- Application log - these contain the usual error and warnings (i.e. whatever we send to log4j). The default location, as you found is /var/log and as you discovered, there's no good way to tell Cloudera Manager that you are now storing them in a different location and to monitor space accordingly.  I'll open a bug for this and will try to resolve this in the next release.

Highlighted
New Contributor
Posts: 1
Registered: ‎02-02-2016

Re: Kafka log directory warning

I have customer with the same issue and they are using CDH 5.4.7. The new version of CDH have moved the logs from / or still the same ? thanks
Announcements

Currently incubating in Cloudera Labs:

Envelope
HTrace
Ibis
Impyla
Livy
Oryx
Phoenix
Spark Runner for Beam SDK
Time Series for Spark
YCSB