Atomic update of Solr document via morphlineSolrSink possible ?

geko — Fri, 16 Sep 2022 08:47:13 GMT

Hi,

is it possible to configure the morphline.conf, so that the loadSolr command can perform an atomic update of a field in the document ?

Why ?

I want to do something like:

- logfile analysis via flume using the morphlineSolrSink to store the output in a Solr doc

- apply some grok regex patterns in the morphline

- if match, then create a field with the fieldname consisting of user-id+request and the value should be the count of how often this user fired that request, thereby just increment this value on any matched morphline pipeline.

- afterwards the document is stored in Solr with the created field as unique identifier

I assume the loadSolr command implicitly calls an update on the solr server and since Solr support atomic update to e.g. increment a counter, it would be great to be able to use this feature inside a morphlineSolrSink.

any help appreciated....Gerd....

Re: Atomic update of Solr document via morphlineSolrSink possible ?

whosch — Fri, 23 Aug 2013 22:09:52 GMT

The loadSolr command doesn't support atomic updates, but you could write a custom morphline command that implements something like this.

Re: Atomic update of Solr document via morphlineSolrSink possible ?

geko — Mon, 26 Aug 2013 06:40:43 GMT

ah, ok. Interesting topic, seems like I have to check this out.....

regards.....Gerd....

question Atomic update of Solr document via morphlineSolrSink possible ? in Archives of Support Questions (Read Only)

Atomic update of Solr document via morphlineSolrSink possible ?

Re: Atomic update of Solr document via morphlineSolrSink possible ?

Re: Atomic update of Solr document via morphlineSolrSink possible ?