Reply
Highlighted
PNS
Explorer
Posts: 38
Registered: ‎05-20-2014
Accepted Solution

Chaining morphlines

Hi...

 

What is the recommended way of chaining independent morphlines together?

 

For example, assume there are multiple such morphlines, each capable of running on its own to implement a well-defined function, and the Java application determines that two or more such functions are successively needed end-to-end (e.g., one to read froma data source and the next to extract the values from the input).

 

One approach of course would be to execute each morphline from within Java and pass the record to the next, until the chain is completed.

 

Is there another way to do so (e.g., somehow compiling together the independent conf files)?

 

 

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Chaining morphlines

Arrange it such that morphlineA pipe records into morphlineB:

 

Command morphlineB = new Compiler().compile(morphlineFileB, morphlineIdB, morphlineContextB, null);
Command morphlineA = new Compiler().compile(morphlineFileA, morphlineIdA, morphlineContextA, morphlineB);

PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Re: Chaining morphlines

Yes, that seems to be the best strategy.

 

I was experimenting with putting the text from different morphline files, each containing a single morphline, via string substitution (e.g., using the Apache Commons Lang StrSubstitutor class).

 

That approach does not work if you use text substitution in the morphline file, because in that case the "declarations' have to be outside the morphline, at the top of the file, so when the files are merged compilation fails.

 

It would have been nice if HOCON supported a C-style #include command. :-)

 

Thanks for the advice!

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Chaining morphlines

PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Re: Chaining morphlines

This does not work for merging with text substitution for the same reason I mentioned in my previous comment, but you are right, it is include functionality.

 

I was probably biased towards the use case for which I created this thread. :-)

 

Thanks.

Announcements
The Kite SDK is a collection of docs, sample code, APIs, and tools to make Hadoop application development faster. Learn more at http://kitesdk.org.