Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Dynamic morphline parameterization using Java

Dynamic morphline parameterization using Java

Explorer

Is there any way (and example) of dynamically changing the fields of a morphline, using Java code?

 

A classic scenario for this would be to define certain values (e.g., constants) in a Java class and use them in one or more places in the morphline.conf file. For example, let's take the case of a grok expression, like:

 

                        {
                            commands : [
                                {
                                    grok {
                                        dictionaryFiles : [.../grok-dictionaries]
                                        expressions : {
                                            message : """%{EXP1:name1} %{EXP2:name2} %{EXP3:name3}"""
                                        }
                                    }
                                }
}

 

How can one define the above fields (name1, name2, name3) in Java and then use placeholders in the morphline file (e.h., %{name1}, %{name2}, %{name3})?

 

5 REPLIES 5

Re: Dynamic morphline parameterization using Java

Expert Contributor
You can define a limited form of variable substitution in HOCON per https://github.com/typesafehub/config/blob/master/HOCON.md but such variables are resolved on morphline compilation and can?t be dynamically changed in Java.

If you want to use Java and be more expressive you can write your own command in Java and have it do whatever you like.

Wolfgang.

Re: Dynamic morphline parameterization using Java

Explorer

I had played wtih variable substitution before posting my question and actually this is what I am trying to achieve: define variables from a Java source.

 

"Pure" Java is certainly an option, but then it is not morphlines. :-)

 

It's surprising that such a feature was is not offered by Typesafe, as part of HOCON.

Re: Dynamic morphline parameterization using Java

Expert Contributor
It would be a custom morphline command, implemented in Java.

Re: Dynamic morphline parameterization using Java

Explorer

I could inject a Java Morphlines Command in an executing pipeline, if there is any way of doing so. Such a Command would intercept the Grok expression, change the field names and instantiate a GrokBuilder to hand over execution.

 

Alternatively, I could use some static mapping in the CommandBuilder that processes the expression, to replace the morphline file fields with the ones defined in Java, but this kind of defeats the purpose.

 

If there was any way of accessing the configuration object from within the Java embedded code of a morphline, that could also provide a solution.

 

Or if there is another way, please suggest. :-)

 

Thanks.

 

Re: Dynamic morphline parameterization using Java

Expert Contributor
Yes, see Section "Implementing your own Custom Command? near the top of http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html

Wolfgang.