Support Questions
Find answers, ask questions, and share your expertise

How to load data from local system file to HDFS using Nifi

I try to load data from my local system file into HDFS by using Getfile processor in Nifi, but it produces "invalid directory" error11252-screenshot-from-2017-01-09-155052.png

1 ACCEPTED SOLUTION

Accepted Solutions

Master Guru
@Narasimma varman

Is your NiFi a single instance of NiFi or a NiFi cluster?

If it is a cluster, keep in my mind by default the GetFile processor will be running on every node in that cluster. The validate will also run on every node as well, so make sure the directory exists on all nodes.

Also make sure you have only specified a d directory path in the "Input Directory" property in GetFile. In you case, you should have only "/root/example" for that property.

The filename you wish to pickup should be specified in the "File Filter" property.

Matt

View solution in original post

7 REPLIES 7

Master Guru
@Narasimma varman

Is your NiFi a single instance of NiFi or a NiFi cluster?

If it is a cluster, keep in my mind by default the GetFile processor will be running on every node in that cluster. The validate will also run on every node as well, so make sure the directory exists on all nodes.

Also make sure you have only specified a d directory path in the "Input Directory" property in GetFile. In you case, you should have only "/root/example" for that property.

The filename you wish to pickup should be specified in the "File Filter" property.

Matt

View solution in original post

Yes, I am using single instance of Nifi

11316-nifi-flow.jpg

Properties Nifi GetFile processor

Super Guru

I see that "Keep Source File" is set to false, if this processor has executed successfully once, then the file will be removed, and when it runs again, the file will no longer be there. Also since the File Filter is a regular expression, you may want to specify sample\.txt (note the backslash) to match the filename exactly (otherwise the period matches any character so a file called sample1txt would be found as well).

Thanks, now I can load from local

Master Guru

@Narasimma varman

Make sure the user the NiFi process is running as on your server has the necessary permissions to access that directory path and remove files from it. Matt

Thanks, now I can load from local