Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi error with PutElasticsearch

Solved Go to solution
Highlighted

Nifi error with PutElasticsearch

New Contributor

I have a simple JSON file that i am trying to index using elastic search.

Here is my JSON file:

{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}

I have elastic search running on my local machine.

Localhost:9200

{
  "name" : "node-01",
  "cluster_name" : "lol_es",
  "version" : {
    "number" : "2.3.1",
    "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp" : "2016-04-04T12:25:05Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

I am using two Nifi processor GetFile (getting the json file) and PutElastic (trying to index it to elastic search)

I get the following error in Nifi

09:41:57 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute ${filename} for StandardFlowFileRecord[uuid=0ced3305-469b-4474-aa26-d14190f2fb2b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1236, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure
09:41:57 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]

Here is my configuration for PutElastic processor

ElasticSearchHost : localhost:9200

IdentifierAttribute: ${filename}

Please let me know what i am missing. Appreciate your help.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi error with PutElasticsearch

Explorer

Hello @Chakra Sankaraiah

As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300.

So, your ElasticSearch Hosts: localhost:9300

And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as:

Identifier Attribute: uuid

Let me know if this helps!.

Thanks,

Amarnath

View solution in original post

6 REPLIES 6
Highlighted

Re: Nifi error with PutElasticsearch

Hi Chakra,

In fact the identifier attribute only requires the name of the attribute you want to reference and not the value of the attribute itself.

I think that if you replace ${filename} by filename, it will work as you expect.

Hope that helps

Highlighted

Re: Nifi error with PutElasticsearch

New Contributor

I tried to change the flow but still get the same two errors

Auto refresh started
10:07:10 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute account1_rec.json for StandardFlowFileRecord[uuid=47d795d9-1aa0-4aad-a197-a5811708685b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1442, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure
10:07:10 CDT ERROR d3887b11-5c51-4ab3-b520-1c52a6fb6204

PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]

Highlighted

Re: Nifi error with PutElasticsearch

Could you check that the issue is not on Elasticsearch side by sending your JSON message manually? (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)

curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'
Highlighted

Re: Nifi error with PutElasticsearch

New Contributor

I have tried the the curl for the file that i am using and it works fine

curl -XPOST http://127.0.0.1:9200/bankman/accountman/1 -d @account1_rec.json {"_index":"bankman","_type":"accountman","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

Highlighted

Re: Nifi error with PutElasticsearch

I am sorry... I didn't look carefully the first time...

In NiFi we are using the 'transport' port (9300 by default) to exchange with elasticsearch and not the 'http' port (9200 by default). Can you check if it is working with port 9300?

Re: Nifi error with PutElasticsearch

Explorer

Hello @Chakra Sankaraiah

As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300.

So, your ElasticSearch Hosts: localhost:9300

And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as:

Identifier Attribute: uuid

Let me know if this helps!.

Thanks,

Amarnath

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here