Reply
PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Dynamic morphline parameterization using Java

[ Edited ]

Is there any way (and example) of dynamically changing the fields of a morphline, using Java code?

 

A classic scenario for this would be to define certain values (e.g., constants) in a Java class and use them in one or more places in the morphline.conf file. For example, let's take the case of a grok expression, like:

 

                        {
                            commands : [
                                {
                                    grok {
                                        dictionaryFiles : [.../grok-dictionaries]
                                        expressions : {
                                            message : """%{EXP1:name1} %{EXP2:name2} %{EXP3:name3}"""
                                        }
                                    }
                                }
}

 

How can one define the above fields (name1, name2, name3) in Java and then use placeholders in the morphline file (e.h., %{name1}, %{name2}, %{name3})?

 

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Dynamic morphline parameterization using Java

You can define a limited form of variable substitution in HOCON per https://github.com/typesafehub/config/blob/master/HOCON.md but such variables are resolved on morphline compilation and can?t be dynamically changed in Java.

If you want to use Java and be more expressive you can write your own command in Java and have it do whatever you like.

Wolfgang.

PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Re: Dynamic morphline parameterization using Java

[ Edited ]

I had played wtih variable substitution before posting my question and actually this is what I am trying to achieve: define variables from a Java source.

 

"Pure" Java is certainly an option, but then it is not morphlines. :-)

 

It's surprising that such a feature was is not offered by Typesafe, as part of HOCON.

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Dynamic morphline parameterization using Java

It would be a custom morphline command, implemented in Java.

PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Re: Dynamic morphline parameterization using Java

[ Edited ]

I could inject a Java Morphlines Command in an executing pipeline, if there is any way of doing so. Such a Command would intercept the Grok expression, change the field names and instantiate a GrokBuilder to hand over execution.

 

Alternatively, I could use some static mapping in the CommandBuilder that processes the expression, to replace the morphline file fields with the ones defined in Java, but this kind of defeats the purpose.

 

If there was any way of accessing the configuration object from within the Java embedded code of a morphline, that could also provide a solution.

 

Or if there is another way, please suggest. :-)

 

Thanks.

 

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Dynamic morphline parameterization using Java

Yes, see Section "Implementing your own Custom Command? near the top of http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html

Wolfgang.

Announcements
The Kite SDK is a collection of docs, sample code, APIs, and tools to make Hadoop application development faster. Learn more at http://kitesdk.org.