Support Questions
Find answers, ask questions, and share your expertise

JSON logfiles to Solr

JSON logfiles to Solr

New Contributor

Hi,

I'm tring to read a json log file and insert into solr collection.

logfile is in following format

{"@timestamp": "2017-02-18T02:16:50.496+04:00","message": "hello"}
{"@timestamp": "2017-02-18T02:16:50.496+04:00","message": "hello"}
{ "@timestamp": "2017-02-18T02:16:50.496+04:00","message": "hello"}

I was able to load the file and split by lines. How can i proceed further ?

4 REPLIES 4

Re: JSON logfiles to Solr

Super Guru

I'm not sure if you'll need additional processing of the flow files, but eventually you'll want a PutSolrContentStream processor to send the document(s) to Solr.

Re: JSON logfiles to Solr

New Contributor

Ive used PutSolrContentStream after spliting the logs.. but nothing is going out from PutSolrContentStream.

Also what if i want to select only particular fields from json to be sent to Solr ?

13665-screenshot-from-2017-03-15-18-59-09.png


screenshot-from-2017-03-15-18-59-35.png

Re: JSON logfiles to Solr

Have you tried using the PutSolrContentStream processor?

https://blogs.apache.org/nifi/entry/indexing_tweets_with_nifi_and

Re: JSON logfiles to Solr

Try using the EvaluateJsonPath processor if you want to pull out key attribute value pairs. You can then map those AVPs by creating additional fields in the properties of the PutSolrContentStream processor. The flow NiFi from this marquee post shows the design pattern.

https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.h...