NiFi - create folder and file names based on hostname and date

[#1]

Hi,

Trying to "practically" replace multiple ingestion tools with NiFi. I have Syslog-NG as one these tools storing syslog messages to files based on the hostname (syslog.hostname) extracted from the syslog message and date the message is received. The following shows the Syslog-NG destination configuration:

"

destination d_network {file("/var/log/network/$HOST/$YEAR/$MONTH/$DAY/syslog.log" owner(root) group(root) perm(0666) dir_perm(0777) create_dirs(yes) ); };

"

Syslog-NG would create the folders automatically based on the hostname, year, month and day, and then store the data to a file "syslog.log". The file would be appended if it already exists.

Couple of challenges I faced with NiFi:

1. The regular file append issue (which a patch exists for)

2. The bigger problem is how to automatically create the folders without the need to manually do this for each source, noting that I have 100s of sources.

Suggestions on how to do this on NiFi would be appreciated.

[#2]

Are you trying to replace the functionality of creating the host/date paths and files and/or are you trying to want NiFi to recursively parse the growing directories to get at the underlying syslog.log files?