I have a process that fetches files from an ftp location and processes them HDFS -> hive. At the end of the day I would like to reconcile that I loaded all files on FTP into the hive table. The hive table has a field for filename so I can get the distinct listing of files loaded for that day using the selectHiveQL processor. I tried getting the list of files off FTP from the listFTP processor but it is just queueing up 20 zero byte files. I envisioned being able to just listFTP -> MergeContent to have a text file of all filenames in ftp directory and then somehow compare the results of selectHiveQL and listFTP/mergecontent but mergecontent doesnt even run with the zero byte files input. Any suggestions on how to do this correctly?