Created 04-29-2016 02:53 PM
I have a simple JSON file that i am trying to index using elastic search.
Here is my JSON file:
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
I have elastic search running on my local machine.
Localhost:9200
{ "name" : "node-01", "cluster_name" : "lol_es", "version" : { "number" : "2.3.1", "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39", "build_timestamp" : "2016-04-04T12:25:05Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }
I am using two Nifi processor GetFile (getting the json file) and PutElastic (trying to index it to elastic search)
I get the following error in Nifi
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute ${filename} for StandardFlowFileRecord[uuid=0ced3305-469b-4474-aa26-d14190f2fb2b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1236, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]
Here is my configuration for PutElastic processor
ElasticSearchHost : localhost:9200
IdentifierAttribute: ${filename}
Please let me know what i am missing. Appreciate your help.
Created 05-03-2016 10:14 AM
Hello @Chakra Sankaraiah
As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300.
So, your ElasticSearch Hosts: localhost:9300
And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as:
Identifier Attribute: uuid
Let me know if this helps!.
Thanks,
Amarnath
Created 04-29-2016 02:56 PM
Hi Chakra,
In fact the identifier attribute only requires the name of the attribute you want to reference and not the value of the attribute itself.
I think that if you replace ${filename} by filename, it will work as you expect.
Hope that helps
Created 04-29-2016 03:07 PM
I tried to change the flow but still get the same two errors
Auto refresh startedPutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute account1_rec.json for StandardFlowFileRecord[uuid=47d795d9-1aa0-4aad-a197-a5811708685b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1442, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]
Created 04-29-2016 03:13 PM
Could you check that the issue is not on Elasticsearch side by sending your JSON message manually? (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)
curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{ "user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" }'
Created 04-29-2016 04:19 PM
I have tried the the curl for the file that i am using and it works fine
curl -XPOST http://127.0.0.1:9200/bankman/accountman/1 -d @account1_rec.json {"_index":"bankman","_type":"accountman","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}
Created 04-29-2016 04:27 PM
I am sorry... I didn't look carefully the first time...
In NiFi we are using the 'transport' port (9300 by default) to exchange with elasticsearch and not the 'http' port (9200 by default). Can you check if it is working with port 9300?
Created 05-03-2016 10:14 AM
Hello @Chakra Sankaraiah
As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300.
So, your ElasticSearch Hosts: localhost:9300
And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as:
Identifier Attribute: uuid
Let me know if this helps!.
Thanks,
Amarnath