02-05-2015 04:06 AM
We are evaluating Kafka installation in ou CHD5.3 cluster.
Due to our directory structure we had to move the log.dir to a non-default location (default is /var/log/kafka)
We changed value of log.dir and all dependant values in log4j.properties.
After this change, Cloudera Manager still shows the status of all brokers as bad, because the free space under default log directory is not sufficient. So Monitoring does not seem to read the value of log.dir from log4j-properties of kafka_broker.
Is there a way how to change Kafka log directory so that it is accepted by Monitoring automatically?
02-05-2015 09:27 AM
I'm glad you got the test cluster working :)
For the log directory, Kafka has two types of logs:
- Transaction log - these contain the actual data written to Kafka, so they are critical to the normal operation of the product. We store those in log.dir which defaults to /var/local/kafka/data
- Application log - these contain the usual error and warnings (i.e. whatever we send to log4j). The default location, as you found is /var/log and as you discovered, there's no good way to tell Cloudera Manager that you are now storing them in a different location and to monitor space accordingly. I'll open a bug for this and will try to resolve this in the next release.
Currently incubating in Cloudera Labs:Envelope