question I want to delete attributes after pulling the data from mysql while pushing to the elastic search. in Archives of Support Questions (Read Only)

I want to delete attributes after pulling the data from mysql while pushing to the elastic search.

sunil_devopsian — Wed, 13 Dec 2017 02:35:39 GMT

I am trying using update attribute processor but i am not able to do it..please tell me how to do it

Re: I want to delete attributes after pulling the data from mysql while pushing to the elastic search.

Shu_ashu — Sun, 18 Aug 2019 01:54:21 GMT

@sunil kumar,
In update attributes processor delete attributes expression property expects attributes in regular expressions.

Delete Attributes Expression

Regular expression for attributes to be deleted from flowfiles.
Supports Expression Language: true

If you want to delete id,CompanyName attributes from the flowfile then

Delete Attributes Expression

id|CompanyName

Same way if you want to delete more attrbutes keep them with pipe separated or you can include matching regex that can match all the attributes that you are desired to delete.

Configs:-

id|Company.* //matches with id attribute and all attribute names starting with Company 
id|Company //matches only id,Company attributes only

By taking reference you can change and configure Update Attribute processor as per your needs.

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors.

Re: I want to delete attributes after pulling the data from mysql while pushing to the elastic search.

sunil_devopsian — Sun, 18 Aug 2019 01:54:13 GMT

hey shu thanks for your quick response.

and yes i am trying the same way which you posted but the attributes are not deleting and they are comming into elastic search..

Do i need to change the nifi flow

Here is my nifi flow and update attributes screen

Re: I want to delete attributes after pulling the data from mysql while pushing to the elastic search.

Shu_ashu — Thu, 14 Dec 2017 01:05:37 GMT

@sunil kumar

We can do your case in different methods
Before Put Elastic search processor we need to make the contents of the flowfile without city in the contents of flowfile.

Let’s consider

[{
  "id":
"1",
  "name":
"Michael",
  "city":
"orlando"
},
{
  "id":
"2",
  "name":
"John",
  "city":
"miami"
}]

You are having above json array with 2 records then you are expecting

[{"id":1,"name":"Michael"},{"id":2,"name":"John"}]

i.e without city in the json message.

Method 1:-

For this case if you are using NiFi 1.2+ there are convert record processor which will help you to get this work done very easily.

I am attaching the xml document here and I tested with the above json as input.

nifi12.xml

Method 2:-

If you are using prior to NiFi 1.2

Then you need to use SplitJson processor after ConvertAVROtojson.

Then use Evaluate json Processor with Destination as flowfile attribute

Then Attributes to Json processor to get only the required attributes into your resultant content.

Now put elastic search processor gets only the required json message elements.

Method 3:-

After splitjson use Replace text processor and put your logic to capture the json message execept city element.

I have attached all the flow xml’s, you can download and make sure with method will best fit for your case.

prior-nifi-12.xml

Re: I want to delete attributes after pulling the data from mysql while pushing to the elastic search.

ASG2020 — Mon, 25 Nov 2019 07:41:29 GMT

Hi,

What if I want to delete all attributes present in FlowFile except, id and CompanyName?