Reply
New Contributor
Posts: 2
Registered: ‎06-10-2015

Debug Morphlines configuration

Hi,

 

I have a Flume configuration that reads file from Spool Dir, transform using Interceptors (with Morphlines) and write to HDFS.

 

I have added debug statements in the Morphlines and been tring to debug the configuration. I am running flume from command line with -Dflume switch (please see below for the command that was used).

 

<<<

flume-ng agent --name Agent1 --conf conf --conf-file $conf -Dflume.root.logger=DEBUG,console

>>>

 

However, I don't see any debug statements that are added in Morhphlines getting reported in the console. All flume logging shows INFO statements but not DEBUG statements.

 

Any guidance or inputs would be greatly appreciated.

 

Thank You!

Explorer
Posts: 11
Registered: ‎06-05-2015

Re: Debug Morphlines configuration

[ Edited ]

We're using the standard logging command in morphlines:

  { logInfo { format : "output record: {}", args : ["@{}"] } }

 

Which will appear in your /var/flume-ng/<log file>

 

Have you tried that or the logDebug command?  http://kitesdk.org/docs/current/morphlines/morphlines-reference-guide.html#logTrace

 

David

 

Cloudera Employee
Posts: 277
Registered: ‎01-09-2014

Re: Debug Morphlines configuration

[ Edited ]

The best thing to do to fully debug morphlines and see the data record prior to and after every morphline command is to turn on morphline log4j trace, by adding the following to the flume log4j.properties file in your "conf" directory:

log4j.logger.org.kitesdk.morphline=TRACE

You will then see the flow of data through each command in the morphline.

New Contributor
Posts: 2
Registered: ‎06-10-2015

Re: Debug Morphlines configuration

Thank you David and pdvorak for your inputs! I will give it a shot tomorrow but sounds like that will address my issue.
Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.