Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Dynamic morphline alteration

Explorer

Hi...

 

What would be the best way of changing parts of a morphline at runtime, i.e. after it has been build from the CONF file?

 

A classic use case for this, is applications running many morphlines that differ only in a few commands. For example, a TryRules pipeline with multiple commands, out of which only 1 or 2 get modified every time, depending on the input.

 

In such cases, the "base" morphline can be built with "empty" (no-op) commands as placeholders from the main CONF file and the applicable commands can then be compiled from small CONF files and inserted in the position of the placeholders, before the whole pipeline can be executed.

 

Has anyone done this? Any code examples?

 

Thanks.

 

 

1 ACCEPTED SOLUTION

Expert Contributor
Basically, yes. Beyon see there are a gazillion text templating tools already out there, e.g. freemarker et al. No need to reinvent the wheel.

View solution in original post

4 REPLIES 4

Expert Contributor
Seems like sed or similar on the text of the morphline config file are your friends here.

Explorer

Hmmm...

 

You mean, read the morphline config file, do a text "search and replace" and then compile it?

 

The problem I described (having lots of config code replication) is very common, maybe you should look at providing a more "programatic" solution. 🙂

Expert Contributor
Basically, yes. Beyon see there are a gazillion text templating tools already out there, e.g. freemarker et al. No need to reinvent the wheel.

Explorer

Sure.

 

FreeMarker is a good idea, especially now that they have switched to the Apache 2.0 License.

 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.