I am getting files in batches in my local folder like
1st batch getting 4 files(a.txt, b.log, c.error, d.csv)
After getting all files then after files will transfer to hdfs.
If any files fails, the files will not transfer to hdfs and send email notification to user.
Same for 2nd batch getting two files.
Any ideas or suggestions...