Member since
05-20-2014
38
Posts
0
Kudos Received
0
Solutions
12-09-2014
10:27 AM
This does not work for merging with text substitution for the same reason I mentioned in my previous comment, but you are right, it is include functionality. I was probably biased towards the use case for which I created this thread. 🙂 Thanks.
... View more
12-09-2014
10:01 AM
Yes, that seems to be the best strategy. I was experimenting with putting the text from different morphline files, each containing a single morphline, via string substitution (e.g., using the Apache Commons Lang StrSubstitutor class). That approach does not work if you use text substitution in the morphline file, because in that case the "declarations' have to be outside the morphline, at the top of the file, so when the files are merged compilation fails. It would have been nice if HOCON supported a C-style #include command. 🙂 Thanks for the advice!
... View more
12-09-2014
08:59 AM
Hi... What is the recommended way of chaining independent morphlines together? For example, assume there are multiple such morphlines, each capable of running on its own to implement a well-defined function, and the Java application determines that two or more such functions are successively needed end-to-end (e.g., one to read froma data source and the next to extract the values from the input). One approach of course would be to execute each morphline from within Java and pass the record to the next, until the chain is completed. Is there another way to do so (e.g., somehow compiling together the independent conf files)?
... View more
10-17-2014
01:04 AM
I assumed so, but wanted to confirm that there is no state sharing "behind the scenes", especially in the Command building process. Thanks for the confirmation.
... View more
10-17-2014
01:02 AM
Sure. FreeMarker is a good idea, especially now that they have switched to the Apache 2.0 License.
... View more
10-16-2014
03:30 PM
Hi... Does the single-threaded design of morphilnes mean that there can be concurrency issues by creating multiple morphlines from the same config file and running each of them in a separate thread? For example, if one compiles 10 morphlines from the same config file and then run each of those in a separate thread, can that cause any thread safety issues? Thanks.
... View more
10-16-2014
03:18 PM
Hmmm... You mean, read the morphline config file, do a text "search and replace" and then compile it? The problem I described (having lots of config code replication) is very common, maybe you should look at providing a more "programatic" solution. 🙂
... View more
10-15-2014
11:39 AM
Hi... What would be the best way of changing parts of a morphline at runtime, i.e. after it has been build from the CONF file? A classic use case for this, is applications running many morphlines that differ only in a few commands. For example, a TryRules pipeline with multiple commands, out of which only 1 or 2 get modified every time, depending on the input. In such cases, the "base" morphline can be built with "empty" (no-op) commands as placeholders from the main CONF file and the applicable commands can then be compiled from small CONF files and inserted in the position of the placeholders, before the whole pipeline can be executed. Has anyone done this? Any code examples? Thanks.
... View more
05-29-2014
07:23 AM
Thanks for the prompt answer! Adding to the Morphlines Reference Guide, the tryRules block has to be placed inside the commands block, e.g. morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
commands : [
{
readLine {
charset : UTF-8
}
}
{
tryRules {
catchExceptions : false
throwExceptionIfAllRulesFailed : true
rules : [
# next rule of tryRules cmd:
{
commands : [
{ ... }
{ ... }
]
}
# next rule of tryRules cmd:
{
commands : [
{ ... }
{ ... }
]
}
}
}
]
}
... View more
05-29-2014
04:38 AM
Hi... Is there any way of ignoring an error in a morphline, which consists of a series of commands, without modiying the offending command to be more tolerant? For example, if one of the commands is Grok and a large file is processed and somewhere in the file a record does not match the regular expression, the execution of the morphline stops. Is there any configuration option to allow it to continue? Thanks, PNS
... View more