Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Using Attribute for Output Directory in MoveHDFS processor

avatar
New Contributor

Hello,

I try to use FlowFile Attribute to set output directory in a MoveHDFS processor.

I set the Ouptut Directory to theis value :${path_to_prod_AS}${Librairie}

${path_to_prod_AS} is a variable and ${Librairie} is an Attribute.

My problem is that The file are move in ${path_to_prod_AS} driectory, it's as if he could not interpret the value of the attribute.

For the Input Directory, I Use that string : ${path_to_workspace}${now():format('yyyy-MM-dd')}/${Serveur}/${Librairie}/${Table}/

where ${path_to_workspace} is a variable and ${Serveur} ,${Librairie} and ${Table} ar attribute. There is no problem to collect the file.

Thank you for your help

1 ACCEPTED SOLUTION

avatar
Master Guru
@Frederic Le Texier

The issue is with the attribute ${Librairie} in Output Directory property value.

As per NiFi-1.7 MoveHDFS processor Usage

85516-movehdfs.png

OutputDirectory Property value supports Expression Language but evaluated using variable registry only, But for InputDirectory/File property value will be evaluated using flowfile attributes and variable registry.

To resolve this issue:
Define your ${Librairie} attribute value in variable registry if possible.

As right now you are having the variable as flowfile attribute which this value will not be evaluated and files are moving to ${path_to_prod_AS} directory.

View solution in original post

4 REPLIES 4

avatar
Master Guru
@Frederic Le Texier

The issue is with the attribute ${Librairie} in Output Directory property value.

As per NiFi-1.7 MoveHDFS processor Usage

85516-movehdfs.png

OutputDirectory Property value supports Expression Language but evaluated using variable registry only, But for InputDirectory/File property value will be evaluated using flowfile attributes and variable registry.

To resolve this issue:
Define your ${Librairie} attribute value in variable registry if possible.

As right now you are having the variable as flowfile attribute which this value will not be evaluated and files are moving to ${path_to_prod_AS} directory.

avatar
Master Guru
@Frederic Le Texier

I'm looking into NiFi-1.7.1 documentation and the source code link of MoveHDFS processor(which have updated documentation)

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/...

In apache nifi org the documentation is created based NiFi-1.5 version, probably these missing things are updated NiFi 1.5+ versions.

As you have proposed below, we need to use RouteOnAttribute processor to make decision based on the attribute.

avatar
New Contributor

Thank's a lot for your answer @Shu,

Can you tell me where did you find this information ? I was on the apache site and here is what I could find :

85541-help-movehdfs-apache.png

My problem is that this attribute depends on the flowfile and I wish I could change the final directory depending on the value of this attribute.

I have the impression that it is not possible to change the value of a variable dynamically. The only solution I see is to route the flow based on the attribute, but it will generate me a lot of branch in my process. Do you think there would be a more dynamic solution?

avatar
Rising Star

Any progress on getting MoveHDFS to accept attributes in Output Directory? It seems difficult not be able to have a dynamic solution as mentioned in this thread.