Reply
Expert Contributor
Posts: 139
Registered: ‎07-21-2014

How to test morphlines

I'm looking into constructing a Flume morphlines interceptor and came across this sample code to test the morphlines.conf against sample records:

 

 https://github.com/cloudera/cdk/blob/master/cdk-morphlines/cdk-morphlines-core/src/test/java/com/clo...

 

But when I clone the cdk git repo and run 'mvn clean build' I get this error:

 

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce (clean) on project cdk-parent: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]

 

I'm new to the mvn world and hopefully someone can point me to the right instructions to build the cdk project. Thanks!

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: How to test morphlines

The CDK has been renamed to Kite and instructions how to build it are here:

https://github.com/kite-sdk/kite/

 

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: How to test morphlines

Expert Contributor
Posts: 139
Registered: ‎07-21-2014

Re: How to test morphlines

Thanks, I was able to build kite-examples-morphlines after updating the kite version to 0.17.1-SNAPSHOT.

 

How can we look at the output record when I run this test?

 

mvn test -DskipTests exec:java -Dexec.mainClass="org.kitesdk.morphline.api.MorphlineDemo" -Dexec.args="src/test/resources/test-morphlines/addValues.conf src/test/resources/test-documents/email.txt" -Dexec.classpathScope=test

 

Since I only notice this with Trace enabled but not the actual output record:

 

[org.kitesdk.morphline.api.MorphlineDemo.main()] DEBUG org.kitesdk.morphline.stdlib.LogDebugBuilder$LogDebug  - output record: [{_attachment_body=[java.io.BufferedInputStream@29b41961], names=[@{first_name}], pids=[456, hello], source_host=[123], source_type=[text/log, text/log2]}]

 

 

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: How to test morphlines

To automatically print diagnostic information such as the content of records as they pass through the morphline commands, consider enabling TRACE log level, for example by adding the following line to your log4j.properties file:

 

log4j.logger.org.kitesdk.morphline=TRACE

 

Also see http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#/logTrace

Expert Contributor
Posts: 139
Registered: ‎07-21-2014

Re: How to test morphlines

As mentioned in the previous comment, I did already enbled TRACE but did not find anything related to the output records in the stdout except for this:

 

1588 [org.kitesdk.morphline.api.MorphlineDemo.main()] DEBUG org.kitesdk.morphline.stdlib.LogDebugBuilder$LogDebug  - output record: [{_attachment_body=[java.io.BufferedInputStream@5cde2dcb], names=[@{first_name}], pids=[456, hello], source_host=[123], source_type=[text/log, text/log2]}]

 

Please let me know how to look at the resulting transformation on the sample input records? Thanks!

Announcements
The Kite SDK is a collection of docs, sample code, APIs, and tools to make Hadoop application development faster. Learn more at http://kitesdk.org.