NiFi - how to declare and read global properties? (Get File, Consume Kafka)

[#1]

Hi,

I am trying to make my downstream Process Group read from another one, where I would like to store some parameters (let's say in UpdateAttribute processor)

The problem is that my 2nd Process Group has two flows that start with processors that do not accept upstream connections (Get File, Consume Kafka).

Challenge is - how to make Get File read Input Directory property from, lets say, ${dirName} declared somewhere upstream (is it possible?) or globally? How to make Consume Kafka read Topic Name from ${topicName} declared somewhere upstream or globally? etc.

Is there any processor when I can declare some global parameters without having to connect it with the other Process Groups?

Thanks, Michal

[#2 · Accepted solution]

Hi @Michal R,

I'd recommend to have a look at the variable registry:

https://community.hortonworks.com/articles/57304/supporting-custom-properties-for-expression-langua....

Please note that this is only for processor properties supporting expression language, and that, at the moment, properties are loaded when NiFi starts. You cannot change the values while NiFi is running but variable registry will evolve in the near future.

Hope this helps.

[#3]

@Pierre Villard thanks for the tip - that will probably do the job for some of the properties. The problem is that the upstream PG ("A") with all this configuration details is supposed to be re-usable (so that the downstream PG ("B") would remain static - it would just read the parameters). I mean, ideally I would like to have different A's that go to B: A1 -> B, A2 -> B, ... running at the same time. The difference between A1 and A2 would be that A1 has Input Dir = /foo/bar and A2 Input Dir = /brown/fox etc. So in general instances of PG "A" would only be a store of arguments for a call of "B" PG.

[#4]

@Pierre Villard OK, thanks a lot for the link once again - the article was very useful, although it's a shame that changes to custom properties are made just as NiFi gets restarted. This limits the use cases for this functionality.

[#5]

Michal

The NiFi's convention is to look at attribute -> system-property -> env-variable - in that order. Basically what it means that if it can't find attribute in the FlowFile it will check for system property with the same name and if it can't find one it goes to env-variable etc.

Is that what you were looking for? Basically you can declare sys-prop or env-variable and have inbound processors like ConsumeKafka or GetFile to be more dynamic.

Cheers

Oleg

[#6]

@ozhurakousky thanks for the tip - I rather need something that I could flexibly change without having to restart NiFi. It looks like the only way to provide properties to ConsumeKafka or GetFile is via system properties. I might need to find some workaround for that.