question Re: Nifi PutHDFS prepend filename with current date in Archives of Support Questions (Read Only)

Nifi PutHDFS prepend filename with current date

foivos — Fri, 28 Jul 2017 21:30:03 GMT

Hello,

I am trying to read an API via NiFi at 30secs intervals and place the files to HDFS with filename.json prepended with the current date at each moment ('yyyy-mm-dd HH:mm:ss) but no success. I used GetHTTP processor along with UpdateAttribute to change the filename using expression language.

My attempt for that would be

${filename: prepend(${now():format("yyyy-MM-dd HH:mm:ss")})}

however i get a 'Relative path in absolute URI error' and the relationship fails. Otherwise if i remove the filename prepend operation everything works normal and the PutHDFS processor places the files in the specified folder correctly. Really appreciate any help, thanks in advance!

Re: Nifi PutHDFS prepend filename with current date

Wynner — Fri, 28 Jul 2017 22:01:21 GMT

@Foivos A

You're seeing that error because there is a blank space in the file name, try doing this:

${filename: prepend(${now():format("yyyy-MM-dd-HH:mm:ss")})}

Re: Nifi PutHDFS prepend filename with current date

foivos — Tue, 01 Aug 2017 16:10:56 GMT

@Wynner

Thanks for the suggestion, however i still get the same error with this expression

${filename: prepend(${now():format("yyyy-MM-dd-HH:mm:ss")})}

ERROR PutHDFS Failed to write to HDFS due to java.lang.IllegalArgumentException: java.net. URISyntaxException:

Relative path in absolute URI: .2017-08-01-11:01:13filename.json

I am not sure where this error comes from to be honest, plus there is a dot (.) before the date in the created filename in the error message that should not be there according to the expression used to prepend the filename.

Again, the flow is GetHttps --> UpdateAttribute --> PutHDFS and without any filename manipulations in the 2nd step everything works as it should. Any help is greately appreciated, thanks in advance!

Regards,

Foivos

Re: Nifi PutHDFS prepend filename with current date

dchaffey — Tue, 01 Aug 2017 21:25:33 GMT

I'm pretty sure colons in filenames are considered bad practice - try removing that.

The dot could be because it's failing to complete the writing of the file, so it's generating the dot file for writing but can't complete the action. Just a guess without looking at your logs.

Re: Nifi PutHDFS prepend filename with current date

foivos — Thu, 03 Aug 2017 21:35:41 GMT

Thanks, colons were indeed the problem!