Hi, Please find the answers below: 1. When sending files to hadoop, the files in the spool are not moved anywhere, which makes me wonder if there is a new file in the spool, how does Flume recognize the old and new files? Ans: The files get renamed and a suffix is added to the completely ingested file from spool dir, see the following configuration: fileSuffix .COMPLETED Suffix to append to completely ingested files 2. How does Flume after uploading the file to hadoop, will the files in the spool be moved to another folder? Or does Flume have a mechanism to back up files? Ans: Same as above, It is renamed with suffix. 3. I know that Flume has some properties that help work with regex, but I don't know if Flume supports sending files to hadoop and sorting those files into regex-based directories? If so, how do I do it? Ans: You can use the HDFS directory path with certain formatting escape sequences that will replaced by the HDFS sink to generate a directory/file name to store the events. For example to store the file in different directory based on dates hdfs.path = /flume/%Y-%m-%d For more detail on the escape sequence see the following link: https://flume.apache.org/FlumeUserGuide.html#hdfs-sink 4. Does Flume support sending files to hadoop and categorizing them into directories based on the date sent? (I have read that part in HDFS Sink but when I tried it failed) Ans: - If you give the configuration you are using , I can try to fix the issues with it. 5. While using Flume to send files to hadoop, can I fix the file contents such as adding file names into the data stream, or changing the ";" into "|"? Ans: If you just want to add the file name to the data, you should try following configuration for the spplodir source type: basenameHeader false Whether to add a header storing the basename of the file. basenameHeaderKey basename Header Key to use when appending basename of file to event header. If you want to do regex replace , you will have to use Search and Replace Interceptor You can specify the search regex and replace string. See the following link: https://flume.apache.org/FlumeUserGuide.html#search-and-replace-interceptor 6. Can I use any API, or any tool to monitor Flume file transfer to hadoop? For example, during file transfer, see how many files have been transferred to hadoop or how many files have been successfully submitted and how many files sent to hadoop failed. Ans: Not sure if anything available for spooldir, but you should see the Monitoring section and see if you can use something https://flume.apache.org/FlumeUserGuide.html#monitoring 7. Does Flume record transaction logs with hadoop? For example, how many files have been uploaded to hadoop, ... Ans: I don't think so but might need more research to see if you can track what all files has been written. You can check your spool dir for files sent. Thanks Bimal
... View more
