08-22-2013 10:24 AM
is it possible to configure the morphline.conf, so that the loadSolr command can perform an atomic update of a field in the document ?
I want to do something like:
- logfile analysis via flume using the morphlineSolrSink to store the output in a Solr doc
- apply some grok regex patterns in the morphline
- if match, then create a field with the fieldname consisting of user-id+request and the value should be the count of how often this user fired that request, thereby just increment this value on any matched morphline pipeline.
- afterwards the document is stored in Solr with the created field as unique identifier
I assume the loadSolr command implicitly calls an update on the solr server and since Solr support atomic update to e.g. increment a counter, it would be great to be able to use this feature inside a morphlineSolrSink.
any help appreciated....Gerd....
08-23-2013 03:09 PM
The loadSolr command doesn't support atomic updates, but you could write a custom morphline command that implements something like this.