Support Questions
Find answers, ask questions, and share your expertise

Copy file from Remote server and transfer it to HDFS using NiFi

Copy file from Remote server and transfer it to HDFS using NiFi

Contributor

I have a use case to implement : Copy a file on remote server to HDFS using NiFi. I thought to do it by using GetFTP processor at input end and PutHDFS processor at output end.

But here is a challenge. I need to open FTP port of SOURCE server to read files.

Is there any other way to do this use case in NiFi ?

1 REPLY 1
Highlighted

Re: Copy file from Remote server and transfer it to HDFS using NiFi

@Pradhuman Gupta

I don't understand your problem. You said you thought you would use the GetFTP processor to pull files. This should work fine for you. The processor allows you to specify the port (default to 21) for FTP.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetFTP/

If you want to connect to a server via FTP, then the server has to expose the FTP service. If the server doesn't have FTP enabled and you are not able to enable it for one reason or another, you could configure Apache httpd or nginx web servers to serve up the files and perhaps use the GetHTTP processor.

Another option would be to install NiFi on the server in question and use NiFi Site-to-Site to transfer the files from one NiFi instance to another.