Member since
03-17-2016
17
Posts
4
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9242 | 05-03-2016 10:14 AM |
05-12-2016
04:16 PM
Thank @Massimiliano Nigrelli for the information, it is helpful.
... View more
05-09-2016
02:39 PM
Thanks @Massimiliano Nigrelli , would you mind sharing your flow as a template (.xml), I am looking for configurations for 2 ReplaceText processor's and MergeProcessor. or even pictures would do. Let me know if you need my email.? Regards, Amarnath
... View more
05-09-2016
08:07 AM
Thanks @mburgess and @Simon Elliston Ball for confirming. Also, on suggested ways (SplitText) to send multiple documents to ES, I'm afraid that multiple flow files / events gets created and it is too much on a Raspberry Pi. My original files are each 1.1 MB and at max 10 such files can exists. I tried http post (documents formatted as per ES bulk api) it works fine, however processing and converting to the required format takes a lot of time on pi (using executeScript in python), So I am going with a script and ES client in python (No Nifi) to process and write to ES.
... View more
05-03-2016
02:44 PM
Thanks @drussell for your response. I wish other community members could confirm this.
... View more
05-03-2016
02:18 PM
Hello @drussell I know that putElasticsearch does write contents of mutile flowfile's at a time to ES driven by Batch size attribute. where each flowfile contain just 1 record, i.e., one single json record. But Let me refine my words: I am looking to have a multiple flowfile, where each flowfile contain mutile records unlike in the above case (in this format - (action_and_meta_data\n optional_source\n). As shown in my question. Is it clear what I am looking at? Regards, Amarnath
... View more
05-03-2016
01:44 PM
1 Kudo
Hello All, I want to write to ES using nifi. So I choose to use putElasticsearch processor to do this. I want to know is it possible to do bulk insert to ES (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) using putElasticsearch processor? My understanding: putElasticsearch writes to ES by reading the content of the input flowfile i.e, just one record in json format as shown below. ex: {"priority": "DEBUG", "classname": "ServiceImpl", "message": "Getting node by name XXX", "creationTime": "2016-04-20T15:38:43.000000Z"} However, I want to perform a bulk insert. thus my flowfile contents will be in the format: (as specified by ES bulk api) action_and_meta_data\n
optional_source\n action_and_meta_data\n
optional_source\n
.... For example: {"index": {"_type": "xxxxxlogs", "_id": "2016-04-20-15:13:57-945", "_index": "xxxxlogs-2016-04-20"}} \n {"priority": "DEBUG", "classname": "ServiceImpl", "message": "Getting node by name XXXX", "creationTime": "2016-04-20T15:13:57.000000Z"} \n {"index": {"_type": "xxxxlogs-", "_id": "2016-04-20-15:13:57-941", "_index": "xxxxlogs-2016-04-20"}} \n {"priority": "DEBUG", "classname": "ServiceImpl", "message": "Got node idx XXX", "creationTime": "2016-04-20T15:13:57.000000Z"} \n ....... Thousands of such entries: (action_and_meta_data\n optional_source\n) Please let me know if this can be achieved with putElasticsearch processor? or point me to any specific format for my flowfile conten? Otherwise please let me know how to achieve bulk insert to es using NIFI ? Many thanks. Regards, Amarnath
... View more
Labels:
- Labels:
-
Apache NiFi
05-03-2016
10:14 AM
Hello @Chakra Sankaraiah As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300. So, your ElasticSearch Hosts: localhost:9300 And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as: Identifier Attribute: uuid Let me know if this helps!. Thanks, Amarnath
... View more
04-13-2016
12:51 PM
Hello @Randy Gelhausen Thanks for your information. Indeed it took 30-40 mins for Nifi Web ui to be functional, after excluding all the nar that aren't required for now. I wan't to know if you are using nifi in the clustered environment? Thanks, Amar.
... View more
03-17-2016
01:45 PM
@Randy Gelhausen @Saptak Sen, @Andrew Grande, @pwendorf, @jwitt [just to notify you]
... View more
03-17-2016
11:50 AM
I doing the similar task but web UI is not functioning. I have downloaded NIFI on to my Raspberry Pi 2 and run the ./nifi.sh start command (which is not returning prompt back, I waited for 10-15 mins, then after 1 enter click I get my terminal back to run other commands.). when I run "status" command it says NIFI is running currently. However if I try to stop NIFI, I get "Waiting Apache Nifi to finish shutting down..." then "NIFI has not finished shutting down after 20 seconds. Killing process" and it stops. But web UI is not functioning. NO errors in logs. Can some one please help with this. Thanks, Amar
... View more