Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to load data from local system file to HDFS using Nifi

avatar
Rising Star

I try to load data from my local system file into HDFS by using Getfile processor in Nifi, but it produces "invalid directory" error11252-screenshot-from-2017-01-09-155052.png

1 ACCEPTED SOLUTION

avatar
Super Mentor
@Narasimma varman

Is your NiFi a single instance of NiFi or a NiFi cluster?

If it is a cluster, keep in my mind by default the GetFile processor will be running on every node in that cluster. The validate will also run on every node as well, so make sure the directory exists on all nodes.

Also make sure you have only specified a d directory path in the "Input Directory" property in GetFile. In you case, you should have only "/root/example" for that property.

The filename you wish to pickup should be specified in the "File Filter" property.

Matt

View solution in original post

7 REPLIES 7

avatar
Super Mentor
@Narasimma varman

Is your NiFi a single instance of NiFi or a NiFi cluster?

If it is a cluster, keep in my mind by default the GetFile processor will be running on every node in that cluster. The validate will also run on every node as well, so make sure the directory exists on all nodes.

Also make sure you have only specified a d directory path in the "Input Directory" property in GetFile. In you case, you should have only "/root/example" for that property.

The filename you wish to pickup should be specified in the "File Filter" property.

Matt

avatar
Rising Star

Yes, I am using single instance of Nifi

avatar
Rising Star

11316-nifi-flow.jpg

Properties Nifi GetFile processor

avatar
Master Guru

I see that "Keep Source File" is set to false, if this processor has executed successfully once, then the file will be removed, and when it runs again, the file will no longer be there. Also since the File Filter is a regular expression, you may want to specify sample\.txt (note the backslash) to match the filename exactly (otherwise the period matches any character so a file called sample1txt would be found as well).

avatar
Rising Star

Thanks, now I can load from local

avatar
Super Mentor

@Narasimma varman

Make sure the user the NiFi process is running as on your server has the necessary permissions to access that directory path and remove files from it. Matt

avatar
Rising Star

Thanks, now I can load from local