Nifi error with PutElasticsearch

[#1]

I have a simple JSON file that i am trying to index using elastic search.

Here is my JSON file:

{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}

I have elastic search running on my local machine.

Localhost:9200

{
  "name" : "node-01",
  "cluster_name" : "lol_es",
  "version" : {
    "number" : "2.3.1",
    "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp" : "2016-04-04T12:25:05Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

I am using two Nifi processor GetFile (getting the json file) and PutElastic (trying to index it to elastic search)

I get the following error in Nifi

09:41:57 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204

PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute ${filename} for StandardFlowFileRecord[uuid=0ced3305-469b-4474-aa26-d14190f2fb2b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1236, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure

09:41:57 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204

PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]

Here is my configuration for PutElastic processor

ElasticSearchHost : localhost:9200

IdentifierAttribute: ${filename}

Please let me know what i am missing. Appreciate your help.

[#2]

Hi Chakra,

In fact the identifier attribute only requires the name of the attribute you want to reference and not the value of the attribute itself.

I think that if you replace ${filename} by filename, it will work as you expect.

Hope that helps

[#3]

I tried to change the flow but still get the same two errors

Auto refresh started

10:07:10 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204

PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute account1_rec.json for StandardFlowFileRecord[uuid=47d795d9-1aa0-4aad-a197-a5811708685b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1442, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure

10:07:10 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204

PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]

[#4]

Could you check that the issue is not on Elasticsearch side by sending your JSON message manually? (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)

curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

[#5]

I have tried the the curl for the file that i am using and it works fine

curl -XPOST http://127.0.0.1:9200/bankman/accountman/1 -d @account1_rec.json {"_index":"bankman","_type":"accountman","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

[#6]

I am sorry... I didn't look carefully the first time...

In NiFi we are using the 'transport' port (9300 by default) to exchange with elasticsearch and not the 'http' port (9200 by default). Can you check if it is working with port 9300?

[#7 · Accepted solution]

Hello @Chakra Sankaraiah

As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300.

So, your ElasticSearch Hosts: localhost:9300

And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as:

Identifier Attribute: uuid

Let me know if this helps!.

Thanks,

Amarnath