Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to load data from local system file to HDFS using Nifi

Solved Go to solution

How to load data from local system file to HDFS using Nifi

New Contributor

I try to load data from my local system file into HDFS by using Getfile processor in Nifi, but it produces "invalid directory" error11252-screenshot-from-2017-01-09-155052.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to load data from local system file to HDFS using Nifi

Master Guru
@Narasimma varman

Is your NiFi a single instance of NiFi or a NiFi cluster?

If it is a cluster, keep in my mind by default the GetFile processor will be running on every node in that cluster. The validate will also run on every node as well, so make sure the directory exists on all nodes.

Also make sure you have only specified a d directory path in the "Input Directory" property in GetFile. In you case, you should have only "/root/example" for that property.

The filename you wish to pickup should be specified in the "File Filter" property.

Matt

7 REPLIES 7

Re: How to load data from local system file to HDFS using Nifi

Master Guru
@Narasimma varman

Is your NiFi a single instance of NiFi or a NiFi cluster?

If it is a cluster, keep in my mind by default the GetFile processor will be running on every node in that cluster. The validate will also run on every node as well, so make sure the directory exists on all nodes.

Also make sure you have only specified a d directory path in the "Input Directory" property in GetFile. In you case, you should have only "/root/example" for that property.

The filename you wish to pickup should be specified in the "File Filter" property.

Matt

Highlighted

Re: How to load data from local system file to HDFS using Nifi

New Contributor

Yes, I am using single instance of Nifi

Re: How to load data from local system file to HDFS using Nifi

New Contributor

11316-nifi-flow.jpg

Properties Nifi GetFile processor

Re: How to load data from local system file to HDFS using Nifi

I see that "Keep Source File" is set to false, if this processor has executed successfully once, then the file will be removed, and when it runs again, the file will no longer be there. Also since the File Filter is a regular expression, you may want to specify sample\.txt (note the backslash) to match the filename exactly (otherwise the period matches any character so a file called sample1txt would be found as well).

Re: How to load data from local system file to HDFS using Nifi

New Contributor

Thanks, now I can load from local

Re: How to load data from local system file to HDFS using Nifi

Master Guru

@Narasimma varman

Make sure the user the NiFi process is running as on your server has the necessary permissions to access that directory path and remove files from it. Matt

Re: How to load data from local system file to HDFS using Nifi

New Contributor

Thanks, now I can load from local

Don't have an account?
Coming from Hortonworks? Activate your account here