About PNS

PNS · ‎12-09-2014

This does not work for merging with text substitution for the same reason I mentioned in my previous comment, but you are right, it is include functionality. I was probably biased towards the use case for which I created this thread. 🙂 Thanks.

PNS · ‎12-09-2014

Yes, that seems to be the best strategy. I was experimenting with putting the text from different morphline files, each containing a single morphline, via string substitution (e.g., using the Apache Commons Lang StrSubstitutor class). That approach does not work if you use text substitution in the morphline file, because in that case the "declarations' have to be outside the morphline, at the top of the file, so when the files are merged compilation fails. It would have been nice if HOCON supported a C-style #include command. 🙂 Thanks for the advice!

PNS · ‎12-09-2014

Hi... What is the recommended way of chaining independent morphlines together? For example, assume there are multiple such morphlines, each capable of running on its own to implement a well-defined function, and the Java application determines that two or more such functions are successively needed end-to-end (e.g., one to read froma data source and the next to extract the values from the input). One approach of course would be to execute each morphline from within Java and pass the record to the next, until the chain is completed. Is there another way to do so (e.g., somehow compiling together the independent conf files)?

PNS · ‎10-17-2014

I assumed so, but wanted to confirm that there is no state sharing "behind the scenes", especially in the Command building process. Thanks for the confirmation.

PNS · ‎10-17-2014

Sure. FreeMarker is a good idea, especially now that they have switched to the Apache 2.0 License.

PNS · ‎10-16-2014

Hi... Does the single-threaded design of morphilnes mean that there can be concurrency issues by creating multiple morphlines from the same config file and running each of them in a separate thread? For example, if one compiles 10 morphlines from the same config file and then run each of those in a separate thread, can that cause any thread safety issues? Thanks.

PNS · ‎10-16-2014

Hmmm... You mean, read the morphline config file, do a text "search and replace" and then compile it? The problem I described (having lots of config code replication) is very common, maybe you should look at providing a more "programatic" solution. 🙂

PNS · ‎10-15-2014

Hi... What would be the best way of changing parts of a morphline at runtime, i.e. after it has been build from the CONF file? A classic use case for this, is applications running many morphlines that differ only in a few commands. For example, a TryRules pipeline with multiple commands, out of which only 1 or 2 get modified every time, depending on the input. In such cases, the "base" morphline can be built with "empty" (no-op) commands as placeholders from the main CONF file and the applicable commands can then be compiled from small CONF files and inserted in the position of the placeholders, before the whole pipeline can be executed. Has anyone done this? Any code examples? Thanks.

PNS · ‎05-29-2014

Thanks for the prompt answer! Adding to the Morphlines Reference Guide, the tryRules block has to be placed inside the commands block, e.g. morphlines : [ { id : morphline1 importCommands : ["org.kitesdk.**", "org.apache.solr.**"] commands : [ { readLine { charset : UTF-8 } } { tryRules { catchExceptions : false throwExceptionIfAllRulesFailed : true rules : [ # next rule of tryRules cmd: { commands : [ { ... } { ... } ] } # next rule of tryRules cmd: { commands : [ { ... } { ... } ] } } } ] }

PNS · ‎05-29-2014

Hi... Is there any way of ignoring an error in a morphline, which consists of a series of commands, without modiying the offending command to be more tolerant? For example, if one of the commands is Grok and a large file is processed and somewhere in the file a record does not match the regular expression, the execution of the morphline stops. Is there any configuration option to allow it to continue? Thanks, PNS

Online	Offline
Last Visited	‎04-19-2017 12:49 PM

Date Registered	‎05-20-2014 11:32 PM
Last Visited	‎04-19-2017 12:49 PM
Total Messages Posted	38

Re: Chaining morphlines

Re: Chaining morphlines

Chaining morphlines

Re: Morphlines and multithreading

Re: Dynamic morphline alteration

Morphlines and multithreading

Re: Dynamic morphline alteration

Dynamic morphline alteration

Re: Ignoring a morphline error

Ignoring a morphline error