Reply
Highlighted
PNS
Explorer
Posts: 38
Registered: ‎05-20-2014
Accepted Solution

Dynamic morphline alteration

Hi...

 

What would be the best way of changing parts of a morphline at runtime, i.e. after it has been build from the CONF file?

 

A classic use case for this, is applications running many morphlines that differ only in a few commands. For example, a TryRules pipeline with multiple commands, out of which only 1 or 2 get modified every time, depending on the input.

 

In such cases, the "base" morphline can be built with "empty" (no-op) commands as placeholders from the main CONF file and the applicable commands can then be compiled from small CONF files and inserted in the position of the placeholders, before the whole pipeline can be executed.

 

Has anyone done this? Any code examples?

 

Thanks.

 

 

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Dynamic morphline alteration

Seems like sed or similar on the text of the morphline config file are your friends here.

PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Re: Dynamic morphline alteration

Hmmm...

 

You mean, read the morphline config file, do a text "search and replace" and then compile it?

 

The problem I described (having lots of config code replication) is very common, maybe you should look at providing a more "programatic" solution. :-)

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Dynamic morphline alteration

Basically, yes. Beyon see there are a gazillion text templating tools already out there, e.g. freemarker et al. No need to reinvent the wheel.

PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Re: Dynamic morphline alteration

Sure.

 

FreeMarker is a good idea, especially now that they have switched to the Apache 2.0 License.

 

Announcements
The Kite SDK is a collection of docs, sample code, APIs, and tools to make Hadoop application development faster. Learn more at http://kitesdk.org.