- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Nifi error with PutElasticsearch
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
Created ‎04-29-2016 02:53 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have a simple JSON file that i am trying to index using elastic search.
Here is my JSON file:
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
I have elastic search running on my local machine.
Localhost:9200
{ "name" : "node-01", "cluster_name" : "lol_es", "version" : { "number" : "2.3.1", "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39", "build_timestamp" : "2016-04-04T12:25:05Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }
I am using two Nifi processor GetFile (getting the json file) and PutElastic (trying to index it to elastic search)
I get the following error in Nifi
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute ${filename} for StandardFlowFileRecord[uuid=0ced3305-469b-4474-aa26-d14190f2fb2b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1236, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]
Here is my configuration for PutElastic processor
ElasticSearchHost : localhost:9200
IdentifierAttribute: ${filename}
Please let me know what i am missing. Appreciate your help.
Created ‎05-03-2016 10:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Chakra Sankaraiah
As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300.
So, your ElasticSearch Hosts: localhost:9300
And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as:
Identifier Attribute: uuid
Let me know if this helps!.
Thanks,
Amarnath
Created ‎04-29-2016 02:56 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Chakra,
In fact the identifier attribute only requires the name of the attribute you want to reference and not the value of the attribute itself.
I think that if you replace ${filename} by filename, it will work as you expect.
Hope that helps
Created ‎04-29-2016 03:07 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tried to change the flow but still get the same two errors
Auto refresh startedPutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] No value in identifier attribute account1_rec.json for StandardFlowFileRecord[uuid=47d795d9-1aa0-4aad-a197-a5811708685b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461936514797-1, container=default, section=1], offset=1442, length=206],offset=0,name=account1_rec.json,size=206], transferring to failure
PutElasticsearch[id=d3887b11-5c51-4ab3-b520-1c52a6fb6204] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9200}]]
Created ‎04-29-2016 03:13 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Could you check that the issue is not on Elasticsearch side by sending your JSON message manually? (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)
curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{ "user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" }'
Created ‎04-29-2016 04:19 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have tried the the curl for the file that i am using and it works fine
curl -XPOST http://127.0.0.1:9200/bankman/accountman/1 -d @account1_rec.json {"_index":"bankman","_type":"accountman","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}
Created ‎04-29-2016 04:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am sorry... I didn't look carefully the first time...
In NiFi we are using the 'transport' port (9300 by default) to exchange with elasticsearch and not the 'http' port (9200 by default). Can you check if it is working with port 9300?
Created ‎05-03-2016 10:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Chakra Sankaraiah
As pointed in the previous comment you need to use 9300 as a port number since putElasticsearch processors uses the trasport client which is running on 9300.
So, your ElasticSearch Hosts: localhost:9300
And if have filename property set by GetFile processor, then you should be able to use it (filename) as value for identifier attribute. Otherwise, if you dont care the value of identifier attribute and just want to unique value for each flowfile then simple use uuid of the flow file, as:
Identifier Attribute: uuid
Let me know if this helps!.
Thanks,
Amarnath
