question Re: Where are Nifi attributes written? in Support Questions

Where are Nifi attributes written?

murphy_sean — Wed, 07 Dec 2016 05:10:04 GMT

Some processor's written attributes are readily available within the FIowFile attributes downstream. For example 'executesql.row.count' is populated after ExecuteSQL. I'm not seeing the same behavior with many of the other attributes such as SplitJson. Are we expected to use Groovy script or some other customer process to extract these values? A simple example would be appreciated.

Re: Where are Nifi attributes written?

gkeys — Wed, 07 Dec 2016 05:18:42 GMT

You access all attributes using the expression language, typically to use attributes and their derivation as values in other attributes of a processor:

See this for an excellent overview of attributes, how they change with the lifetime of a flow and how they provide programmatic power in your flows: https://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.1.1/bk_HDF_GettingStarted/content/working-with-attributes.html

If this is what you are looking for, let me know by accepting the answer; else, let me know of any gaps or followup questions.

Re: Where are Nifi attributes written?

mburgess — Wed, 07 Dec 2016 05:31:17 GMT

Each processor is responsible for reading and writing whichever attributes it wants to for the purposes of its processing, and those attributes are available in each processor's documentation. SplitJson for example writes the following attributes to each output flow file:

Name	Description
fragment.identifier	All split FlowFiles produced from the same parent FlowFile will have the same randomly generated UUID added for this attribute
fragment.index	A one-up number that indicates the ordering of the split FlowFiles that were created from a single parent FlowFile
fragment.count	The number of split FlowFiles generated from the parent FlowFile
segment.original.filename	The filename of the parent FlowFile

These were added to NiFi 1.0.0 (HDF 2.0) under NIFI-2632, so if you are using a version of NiFi/HDF before that, that's why you won't see these attributes populated by SplitJson.

Re: Where are Nifi attributes written?

murphy_sean — Mon, 19 Aug 2019 08:21:05 GMT

Thanks for the prompt reply. I've also tried to address this question in item:

https://community.hortonworks.com/questions/68745/nifi-iteration-of-queue-entries-between-processors.html

I'm trying to get similar behavior as can be seen below for 'executesql.row.count' by using a user-defined property (property - iteration : value - ${'fragment.index'} <with and without single quote, set prior to 'SplitJson' or set post 'SplitJson')

I'm only able to get 'No value set' or 'Empty value set' no matter what I try.

The splitjson is very straight forward and successfully builds many json array objects using the expression below.

All I'm attempting to due is keep track of the queue position, so I can act on the last row to post the last transaction date. The attribute 'queue position' would also be of interest, but it also contains no data.

Thanks,

~Sean

Re: Where are Nifi attributes written?

murphy_sean — Wed, 07 Dec 2016 05:55:04 GMT

Ah, I see we're at NiFi version 0.5.1.1.1.2.1-34, so that explains why I am not seeing these attributes.

Re: Where are Nifi attributes written?

murphy_sean — Wed, 07 Dec 2016 06:13:11 GMT

@Matt Burgess You are correct about my being back level for this support. Can anyone suggest a workaround, such as another processor attribute that could do something similar?

Re: Where are Nifi attributes written?

murphy_sean — Fri, 09 Dec 2016 22:25:40 GMT

For those who may be at a back level HDF version as we are a good workaround is to use the SplitContent instead as it utilizes many of the attributes Matt has documented above for the SplitJson processor.