Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Chaining morphlines

Solved Go to solution

Chaining morphlines

Explorer

Hi...

 

What is the recommended way of chaining independent morphlines together?

 

For example, assume there are multiple such morphlines, each capable of running on its own to implement a well-defined function, and the Java application determines that two or more such functions are successively needed end-to-end (e.g., one to read froma data source and the next to extract the values from the input).

 

One approach of course would be to execute each morphline from within Java and pass the record to the next, until the chain is completed.

 

Is there another way to do so (e.g., somehow compiling together the independent conf files)?

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Chaining morphlines

Expert Contributor

Arrange it such that morphlineA pipe records into morphlineB:

 

Command morphlineB = new Compiler().compile(morphlineFileB, morphlineIdB, morphlineContextB, null);
Command morphlineA = new Compiler().compile(morphlineFileA, morphlineIdA, morphlineContextA, morphlineB);

4 REPLIES 4

Re: Chaining morphlines

Expert Contributor

Arrange it such that morphlineA pipe records into morphlineB:

 

Command morphlineB = new Compiler().compile(morphlineFileB, morphlineIdB, morphlineContextB, null);
Command morphlineA = new Compiler().compile(morphlineFileA, morphlineIdA, morphlineContextA, morphlineB);

Highlighted

Re: Chaining morphlines

Explorer

Yes, that seems to be the best strategy.

 

I was experimenting with putting the text from different morphline files, each containing a single morphline, via string substitution (e.g., using the Apache Commons Lang StrSubstitutor class).

 

That approach does not work if you use text substitution in the morphline file, because in that case the "declarations' have to be outside the morphline, at the top of the file, so when the files are merged compilation fails.

 

It would have been nice if HOCON supported a C-style #include command. :-)

 

Thanks for the advice!

Re: Chaining morphlines

Expert Contributor

Re: Chaining morphlines

Explorer

This does not work for merging with text substitution for the same reason I mentioned in my previous comment, but you are right, it is include functionality.

 

I was probably biased towards the use case for which I created this thread. :-)

 

Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here