Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

detect duplicates processor

Solved Go to solution
Highlighted

detect duplicates processor

Hi I tried to remove duplicates

I have a json like below

{

"message":"XXXX",

"id":"1"

}

I want to remove all json with the same id

I try using detect duplicates processor

I try Cashe Entry Identifier ${id} but it's not worked as expected

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: detect duplicates processor

If you want to use ${id} as your cache entry identifier, then you need a flow file attribute called id.

You could use EvaluateJsonPath first with an expression of id = $.id to extract the id from the json into an attribute.

View solution in original post

3 REPLIES 3

Re: detect duplicates processor

@Matt Burgess I will appricate your support

Highlighted

Re: detect duplicates processor

If you want to use ${id} as your cache entry identifier, then you need a flow file attribute called id.

You could use EvaluateJsonPath first with an expression of id = $.id to extract the id from the json into an attribute.

View solution in original post

Highlighted

Re: detect duplicates processor

Thanks @Bryan Bende It's worked for me

Don't have an account?
Coming from Hortonworks? Activate your account here