How to enable TRACE logging for morphlines

[#1]

Hi, 

I'm getting this log statement in my logs: 

WARN org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl: Morphline morphlines.conf@morphline1 failed to process record:

In this article (http://blog.cloudera.com/blog/2013/07/morphlines-the-easy-way-to-build-and-integrate-etl-apps-for-ap... is suggested to increase the log level for morphlines to TRACE so you can see what happens to the message as it moves through the commands.

I didn't find an option to do this in the admin console, and when I was looking for log4j.properties I found one in my /etc/flume-ng/conf dir, but that only contains a configuration for flume.log and it looks like it isn't used.

I am getting a log file with the following name: flume-cmf-flume1-AGENT-localhost.localdomain.log, but I can't find the log4j properties that configures the logging for this.

Any idea's how I can enable TRACE level logging for morphlines? I'm using the CDH 4.4 VM Image.

Thanks a lot!

Kristof.

[#2 · Accepted solution]

To print diagnostic information, such as the content of records as they pass through the morphline commands, consider enabling TRACE log level. For example, you might add the following lines to your log4j.properties file:

log4j.logger.com.cloudera.cdk.morphline=TRACE

In Cloudera Manager 4 this can be done by navigating to Services -> Flume -> Configuration -> View and Edit -> Agent (Default) -> Advanced -> Agent Logging Safety Valve, followed by a restart of the Flume service.

For HBase Near Real Time Indexing it can be done as described near the bottom of this manual page: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Search/latest/Cloudera-Search-User-Gu...

Wolfgang

[#3 · Accepted solution]

To print diagnostic information, such as the content of records as they pass through the morphline commands, consider enabling TRACE log level. For example, you might add the following lines to your log4j.properties file:

log4j.logger.com.cloudera.cdk.morphline=TRACE

In Cloudera Manager 4 this can be done by navigating to Services -> Flume -> Configuration -> View and Edit -> Agent (Default) -> Advanced -> Agent Logging Safety Valve, followed by a restart of the Flume service.

For HBase Near Real Time Indexing it can be done as described near the bottom of this manual page: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Search/latest/Cloudera-Search-User-Gu...

Wolfgang

[#4 · Accepted solution]

Awesome!
Thank you Wolfgang