Hi I have elastic search index in below format. here the fields are populated with null values if the data are not supplied to fields . Is there any way by using nifi we can remove null values contain fields before elastic search index loading Details Sample index if all data populated
{
"_index": "bookdb_index",
"_type": "book",
"_id": "1",
"_score": 0.28168046,
"title": "Elasticsearch: The Definitive Guide",
"summary": "A distibuted real-time search and analytics engine",
"publish_date": "2015-02-07",
"num_reviews": 20,
"publisher": "manning"
} Sample index if all not data populated {
"_index": "bookdb_index",
"_type": "book",
"_id": "1",
"_score": 0.28168046,
"title": "Elasticsearch: The Definitive Guide",
"summary": null,
"publish_date": null,
"num_reviews": 20,
"publisher":null
} Avro Scheam registery {
"namespace": "ingestion",
"type": "record",
"name": "quest",
"fields": [
{
"name": "_type",
"type": "string"
},
{
"name": "_id",
"type": "string"
},
{
"name": "_score",
"type": ["string", "null"]
},
{
"name": "title",
"type": ["string", "null"]
},
{
"name": "summary",
"type": ["string", "null"]
},
{
"name": "publish_date",
"type": ["null", "string"]
},
{
"name": "num_reviews",
"type": ["string", "null"]
},
{
"name": "publisher",
"type": ["string", "null"]
},
] I am using Putelasticsearchhttprecord processor to load data to elasticsearch . In nifi 1.6 there is "Suppress Null Values" feature is available , but i am using nifi 1.5 version , Is there any way to resolve the issue I am trying to change Avro schema registry but fields are populated null values some time and sometimes not. Please help me to resolve the issue
... View more
