Support Questions
Find answers, ask questions, and share your expertise

NiFi - create folder and file names based on hostname and date

New Contributor


Trying to "practically" replace multiple ingestion tools with NiFi. I have Syslog-NG as one these tools storing syslog messages to files based on the hostname (syslog.hostname) extracted from the syslog message and date the message is received. The following shows the Syslog-NG destination configuration:


destination d_network {file("/var/log/network/$HOST/$YEAR/$MONTH/$DAY/syslog.log" owner(root) group(root) perm(0666) dir_perm(0777) create_dirs(yes) ); };


Syslog-NG would create the folders automatically based on the hostname, year, month and day, and then store the data to a file "syslog.log". The file would be appended if it already exists.

Couple of challenges I faced with NiFi:

1. The regular file append issue (which a patch exists for)

2. The bigger problem is how to automatically create the folders without the need to manually do this for each source, noting that I have 100s of sources.

Suggestions on how to do this on NiFi would be appreciated.


Are you trying to replace the functionality of creating the host/date paths and files and/or are you trying to want NiFi to recursively parse the growing directories to get at the underlying syslog.log files?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.