Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi - how to declare and read global properties? (Get File, Consume Kafka)

Solved Go to solution
Highlighted

NiFi - how to declare and read global properties? (Get File, Consume Kafka)

Hi,

I am trying to make my downstream Process Group read from another one, where I would like to store some parameters (let's say in UpdateAttribute processor)

The problem is that my 2nd Process Group has two flows that start with processors that do not accept upstream connections (Get File, Consume Kafka).

Challenge is - how to make Get File read Input Directory property from, lets say, ${dirName} declared somewhere upstream (is it possible?) or globally? How to make Consume Kafka read Topic Name from ${topicName} declared somewhere upstream or globally? etc.

Is there any processor when I can declare some global parameters without having to connect it with the other Process Groups?

Thanks, Michal

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi - how to declare and read global properties? (Get File, Consume Kafka)

Hi @Michal R,

I'd recommend to have a look at the variable registry:

https://community.hortonworks.com/articles/57304/supporting-custom-properties-for-expression-langua....

Please note that this is only for processor properties supporting expression language, and that, at the moment, properties are loaded when NiFi starts. You cannot change the values while NiFi is running but variable registry will evolve in the near future.

Hope this helps.

View solution in original post

5 REPLIES 5
Highlighted

Re: NiFi - how to declare and read global properties? (Get File, Consume Kafka)

Hi @Michal R,

I'd recommend to have a look at the variable registry:

https://community.hortonworks.com/articles/57304/supporting-custom-properties-for-expression-langua....

Please note that this is only for processor properties supporting expression language, and that, at the moment, properties are loaded when NiFi starts. You cannot change the values while NiFi is running but variable registry will evolve in the near future.

Hope this helps.

View solution in original post

Highlighted

Re: NiFi - how to declare and read global properties? (Get File, Consume Kafka)

@Pierre Villard thanks for the tip - that will probably do the job for some of the properties. The problem is that the upstream PG ("A") with all this configuration details is supposed to be re-usable (so that the downstream PG ("B") would remain static - it would just read the parameters). I mean, ideally I would like to have different A's that go to B: A1 -> B, A2 -> B, ... running at the same time. The difference between A1 and A2 would be that A1 has Input Dir = /foo/bar and A2 Input Dir = /brown/fox etc. So in general instances of PG "A" would only be a store of arguments for a call of "B" PG.

Highlighted

Re: NiFi - how to declare and read global properties? (Get File, Consume Kafka)

@Pierre Villard OK, thanks a lot for the link once again - the article was very useful, although it's a shame that changes to custom properties are made just as NiFi gets restarted. This limits the use cases for this functionality.

Highlighted

Re: NiFi - how to declare and read global properties? (Get File, Consume Kafka)

Michal

The NiFi's convention is to look at attribute -> system-property -> env-variable - in that order. Basically what it means that if it can't find attribute in the FlowFile it will check for system property with the same name and if it can't find one it goes to env-variable etc.

Is that what you were looking for? Basically you can declare sys-prop or env-variable and have inbound processors like ConsumeKafka or GetFile to be more dynamic.

Cheers

Oleg

Highlighted

Re: NiFi - how to declare and read global properties? (Get File, Consume Kafka)

@ozhurakousky thanks for the tip - I rather need something that I could flexibly change without having to restart NiFi. It looks like the only way to provide properties to ConsumeKafka or GetFile is via system properties. I might need to find some workaround for that.

Don't have an account?
Coming from Hortonworks? Activate your account here