Cloudera Labs
Provide feedback on Cloudera Labs
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka log directory warning

Solved Go to solution
Highlighted

Kafka log directory warning

Explorer

We are evaluating Kafka installation in ou  CHD5.3 cluster.

Due to our directory structure we had to move the log.dir to a non-default location (default is /var/log/kafka)

We changed value of log.dir and all dependant values in log4j.properties.

After this change, Cloudera Manager still shows the status of all brokers as bad, because the free space under default log directory is not sufficient. So Monitoring does not seem to read the value of log.dir from log4j-properties of kafka_broker.

Is there a way how to change Kafka log directory so that it is accepted by Monitoring automatically?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Kafka log directory warning

Contributor

I'm glad you got the test cluster working :)

 

For the log directory, Kafka has two types of logs:

- Transaction log - these contain the actual data written to Kafka, so they are critical to the normal operation of the product. We store those in log.dir which defaults to /var/local/kafka/data

 

- Application log - these contain the usual error and warnings (i.e. whatever we send to log4j). The default location, as you found is /var/log and as you discovered, there's no good way to tell Cloudera Manager that you are now storing them in a different location and to monitor space accordingly.  I'll open a bug for this and will try to resolve this in the next release.

View solution in original post

2 REPLIES 2

Re: Kafka log directory warning

Contributor

I'm glad you got the test cluster working :)

 

For the log directory, Kafka has two types of logs:

- Transaction log - these contain the actual data written to Kafka, so they are critical to the normal operation of the product. We store those in log.dir which defaults to /var/local/kafka/data

 

- Application log - these contain the usual error and warnings (i.e. whatever we send to log4j). The default location, as you found is /var/log and as you discovered, there's no good way to tell Cloudera Manager that you are now storing them in a different location and to monitor space accordingly.  I'll open a bug for this and will try to resolve this in the next release.

View solution in original post

Highlighted

Re: Kafka log directory warning

New Contributor
I have customer with the same issue and they are using CDH 5.4.7. The new version of CDH have moved the logs from / or still the same ? thanks
Don't have an account?
Coming from Hortonworks? Activate your account here