Replacing Custom NiFi Processor for Converting JSON Content to Attributes

Question

Hi Team,I'm currently using a custom NiFi processor to convert JSON file content into flow file attributes, and I'm looking to replace it with a standard NiFi processor if possible.Sample JSON Input:The JSON files look like this (keys may vary per file):{"SourceFilePath": "/root/rdw/SrcFiles","SourceFilePattern": "Sales_data","SourceFile": "Sales_data_20241211.csv"}Current Setup:I'm using a custom processor of type CreateFlatJSON (Bundle: NestedJSONToFlatJSON) to extract the key-value pairs from the JSON and assign them to attributes.After processing, the flow file attributes look like:SourceFilePath = /root/rdw/SrcFilesSourceFile = Sales_data_20231111164536.csvSourceFilePattern = Sales_dataThese attributes are then used in downstream processors like so:${SourceFilePath}/${SourceFile}Current Flow:FetchSFTP → ExtractText → CreateFlatJSON (Custom) → Downstream ProcessorsFetchSFTP: Retrieves the JSON file.ExtractText: Reads the content into the flow file.CreateFlatJSON: Parses JSON and converts key-value pairs to attributes.QuestionIs there a standard NiFi processor that can replace the custom CreateFlatJSON processor and achieve the same functionality — i.e., convert JSON content in the flow file into flow file attributes (with dynamic keys/values)?

linssab · Accepted Answer

You can use a Groovy script with the ExecuteGroovyScript processor.See this answer here for details

MattWho · Answer

@s198

I can't think of any NiFi stock processors that would create dynamic attributes nor can I figure out why you would so this. I understand that you are using these "dynamic" ("${SourceFilePath}/${SourceFile}") attributes downstream in your dataflow, but to do so you are configuring these strings in that downstream processor meaning you need to know what they will be before even starting your processors. If that is the case, these are not really dynamic since you need to know what they are to configure your downstream processors.

If the following strings exist in all your source json, you can just declare them manually with EvaluateJsonPath processor to get the corresponding values from the FlowFile content:

SourceFilePath
SourceFile
SourceFilePattern

Can you share more info about your complete dataflow for better understanding?
What is happening downstream of your custom processor?

Thank you,

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.