Support Questions

Find answers, ask questions, and share your expertise

Using NiFi to load data from localFS to HDFS

avatar
Contributor

I'm trying to load a csv from a 'local FS' into HDFS using NiFi. But I can't seem to figure out what I'm doing wrong.

Based on this thread from @Matt Clarke-

https://community.hortonworks.com/questions/77019/how-to-load-data-from-local-system-file-to-hdfs-us...

I've configured the GetFile and PutHDFS procs as shown-

77405-screen-shot-2018-05-29-at-11444-am.png

And here's the file system-

I feel like there's something simple I'm missing, but I'm struggling with it nonetheless. Can anyone spot where I've gone wrong?

77407-screen-shot-2018-05-29-at-22737-am.png

77406-screen-shot-2018-05-29-at-11541-am.png

10 REPLIES 10

avatar
Master Guru

Are your files still in the source directory? GetFile defaults to removing the files you specify, so if something went wrong you might not have those files anymore, you'd need to set Keep Source File to true in that case (as Matt Clarke recommended). That's why the recommended option is ListFile->FetchFile, as it will keep track of the files it's seen and not fetch those again.