Using Attribute for Output Directory in MoveHDFS processor

[#1]

Hello,

I try to use FlowFile Attribute to set output directory in a MoveHDFS processor.

I set the Ouptut Directory to theis value :${path_to_prod_AS}${Librairie}

${path_to_prod_AS} is a variable and ${Librairie} is an Attribute.

My problem is that The file are move in ${path_to_prod_AS} driectory, it's as if he could not interpret the value of the attribute.

For the Input Directory, I Use that string : ${path_to_workspace}${now():format('yyyy-MM-dd')}/${Serveur}/${Librairie}/${Table}/

where ${path_to_workspace} is a variable and ${Serveur} ,${Librairie} and ${Table} ar attribute. There is no problem to collect the file.

Thank you for your help

[#2 · Accepted solution]

@Frederic Le Texier

The issue is with the attribute ${Librairie} in Output Directory property value.

As per NiFi-1.7 MoveHDFS processor Usage

OutputDirectory Property value supports Expression Language but evaluated using variable registry only, But for InputDirectory/File property value will be evaluated using flowfile attributes and variable registry.

To resolve this issue:
Define your ${Librairie} attribute value in variable registry if possible.

As right now you are having the variable as flowfile attribute which this value will not be evaluated and files are moving to ${path_to_prod_AS} directory.

[#3]

@Frederic Le Texier

I'm looking into NiFi-1.7.1 documentation and the source code link of MoveHDFS processor(which have updated documentation)

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/...

In apache nifi org the documentation is created based NiFi-1.5 version, probably these missing things are updated NiFi 1.5+ versions.

As you have proposed below, we need to use RouteOnAttribute processor to make decision based on the attribute.

[#4]

Thank's a lot for your answer @Shu,

Can you tell me where did you find this information ? I was on the apache site and here is what I could find :

My problem is that this attribute depends on the flowfile and I wish I could change the final directory depending on the value of this attribute.

I have the impression that it is not possible to change the value of a variable dynamically. The only solution I see is to route the flow based on the attribute, but it will generate me a lot of branch in my process. Do you think there would be a more dynamic solution?

[#5]

Any progress on getting MoveHDFS to accept attributes in Output Directory? It seems difficult not be able to have a dynamic solution as mentioned in this thread.