Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Using Attribute for Output Directory in MoveHDFS processor

Solved Go to solution
Highlighted

Using Attribute for Output Directory in MoveHDFS processor

New Contributor

Hello,

I try to use FlowFile Attribute to set output directory in a MoveHDFS processor.

I set the Ouptut Directory to theis value :${path_to_prod_AS}${Librairie}

${path_to_prod_AS} is a variable and ${Librairie} is an Attribute.

My problem is that The file are move in ${path_to_prod_AS} driectory, it's as if he could not interpret the value of the attribute.

For the Input Directory, I Use that string : ${path_to_workspace}${now():format('yyyy-MM-dd')}/${Serveur}/${Librairie}/${Table}/

where ${path_to_workspace} is a variable and ${Serveur} ,${Librairie} and ${Table} ar attribute. There is no problem to collect the file.

Thank you for your help

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Using Attribute for Output Directory in MoveHDFS processor

Super Guru
@Frederic Le Texier

The issue is with the attribute ${Librairie} in Output Directory property value.

As per NiFi-1.7 MoveHDFS processor Usage

85516-movehdfs.png

OutputDirectory Property value supports Expression Language but evaluated using variable registry only, But for InputDirectory/File property value will be evaluated using flowfile attributes and variable registry.

To resolve this issue:
Define your ${Librairie} attribute value in variable registry if possible.

As right now you are having the variable as flowfile attribute which this value will not be evaluated and files are moving to ${path_to_prod_AS} directory.

4 REPLIES 4

Re: Using Attribute for Output Directory in MoveHDFS processor

Super Guru
@Frederic Le Texier

The issue is with the attribute ${Librairie} in Output Directory property value.

As per NiFi-1.7 MoveHDFS processor Usage

85516-movehdfs.png

OutputDirectory Property value supports Expression Language but evaluated using variable registry only, But for InputDirectory/File property value will be evaluated using flowfile attributes and variable registry.

To resolve this issue:
Define your ${Librairie} attribute value in variable registry if possible.

As right now you are having the variable as flowfile attribute which this value will not be evaluated and files are moving to ${path_to_prod_AS} directory.

Re: Using Attribute for Output Directory in MoveHDFS processor

Super Guru
@Frederic Le Texier

I'm looking into NiFi-1.7.1 documentation and the source code link of MoveHDFS processor(which have updated documentation)

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/...

In apache nifi org the documentation is created based NiFi-1.5 version, probably these missing things are updated NiFi 1.5+ versions.

As you have proposed below, we need to use RouteOnAttribute processor to make decision based on the attribute.

Re: Using Attribute for Output Directory in MoveHDFS processor

New Contributor

Thank's a lot for your answer @Shu,

Can you tell me where did you find this information ? I was on the apache site and here is what I could find :

85541-help-movehdfs-apache.png

My problem is that this attribute depends on the flowfile and I wish I could change the final directory depending on the value of this attribute.

I have the impression that it is not possible to change the value of a variable dynamically. The only solution I see is to route the flow based on the attribute, but it will generate me a lot of branch in my process. Do you think there would be a more dynamic solution?

Re: Using Attribute for Output Directory in MoveHDFS processor

New Contributor

Any progress on getting MoveHDFS to accept attributes in Output Directory? It seems difficult not be able to have a dynamic solution as mentioned in this thread.

Don't have an account?
Coming from Hortonworks? Activate your account here