question Re: Using Attribute for Output Directory in MoveHDFS processor in Archives of Support Questions (Read Only)

Using Attribute for Output Directory in MoveHDFS processor

dref278 — Fri, 03 Aug 2018 21:35:29 GMT

Hello,

I try to use FlowFile Attribute to set output directory in a MoveHDFS processor.

I set the Ouptut Directory to theis value :${path_to_prod_AS}${Librairie}

${path_to_prod_AS} is a variable and ${Librairie} is an Attribute.

My problem is that The file are move in ${path_to_prod_AS} driectory, it's as if he could not interpret the value of the attribute.

For the Input Directory, I Use that string : ${path_to_workspace}${now():format('yyyy-MM-dd')}/${Serveur}/${Librairie}/${Table}/

where ${path_to_workspace} is a variable and ${Serveur} ,${Librairie} and ${Table} ar attribute. There is no problem to collect the file.

Thank you for your help

Re: Using Attribute for Output Directory in MoveHDFS processor

Shu_ashu — Sun, 18 Aug 2019 04:00:02 GMT

@Frederic Le Texier

The issue is with the attribute ${Librairie} in Output Directory property value.

As per NiFi-1.7 MoveHDFS processor Usage

OutputDirectory Property value supports Expression Language but evaluated using variable registry only, But for InputDirectory/File property value will be evaluated using flowfile attributes and variable registry.

To resolve this issue:
Define your ${Librairie} attribute value in variable registry if possible.

As right now you are having the variable as flowfile attribute which this value will not be evaluated and files are moving to ${path_to_prod_AS} directory.

Re: Using Attribute for Output Directory in MoveHDFS processor

dref278 — Sun, 18 Aug 2019 03:59:54 GMT

Thank's a lot for your answer @Shu,

Can you tell me where did you find this information ? I was on the apache site and here is what I could find :

My problem is that this attribute depends on the flowfile and I wish I could change the final directory depending on the value of this attribute.

I have the impression that it is not possible to change the value of a variable dynamically. The only solution I see is to route the flow based on the attribute, but it will generate me a lot of branch in my process. Do you think there would be a more dynamic solution?

Re: Using Attribute for Output Directory in MoveHDFS processor

Shu_ashu — Mon, 06 Aug 2018 20:00:04 GMT

@Frederic Le Texier

I'm looking into NiFi-1.7.1 documentation and the source code link of MoveHDFS processor(which have updated documentation)

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/MoveHDFS.java#L150

In apache nifi org the documentation is created based NiFi-1.5 version, probably these missing things are updated NiFi 1.5+ versions.

As you have proposed below, we need to use RouteOnAttribute processor to make decision based on the attribute.

Re: Using Attribute for Output Directory in MoveHDFS processor

henrikolsen — Wed, 09 Jan 2019 17:16:01 GMT

Any progress on getting MoveHDFS to accept attributes in Output Directory? It seems difficult not be able to have a dynamic solution as mentioned in this thread.