Member since
12-10-2018
4
Posts
0
Kudos Received
0
Solutions
12-12-2018
12:27 AM
@Matt Clarke It would be great if this feature could possibly be considered as a future addition. I'm new to the community, so not sure about the process to create a new feature request. Though, if you direct me to the process documentation, I'm happy to create one. Thanks!
... View more
12-11-2018
05:56 PM
@Matt Clarke Thanks for the quick response. Setting filename attribute directly won't scale for us. There could be hundreds of recipients, each with different file name requirement. We could potentially use IfElse, but it'd be highly error prone to maintain hundred+ conditions. More over we already have a tool that captures configuration settings for different recipients and store it in a Sql table, and keeping the Sql table and filename attribute in sync could become a maintenance nightmare. I understand running EL evaluator against a string value could get complicated. The approach I had in mind was a simple one like having an eval function, for example ${filename:eval()}. This function would run an EL evaluator on the value of the attribute. If the attribute value contains a string "${" that is not intended to be an EL statement, it could be escaped. Thanks again! Puneet
... View more
12-11-2018
05:08 AM
Context:
We are using NiFi to read data from a hive store and generate a csv output file. This output file has to be sent out to different recipients, and each recipient has their own requirement for the file name. Some recipients use static name like "enrollments.csv", but others use more dynamic name like "enrollments_<year>_<month>.csv".
We maintain a sql table to store configuration settings for different recipients and file name is one of the settings stored. In the flow, this value is read from the database and assigned to the "filename" attribute, which eventually gets used by PutFile processor when writing content to the disk.
The flow looks something like: ExecuteSQLRecord (Read file name configuration from Sql) --> EvaluateJsonPath (Assign read file name to filename attribute) --> SelectHiveQL (Read data from Hive) --> ConvertRecord (Convert data in Avro format to Csv format) --> PutFile (Writes csv file to disk, using filename attribute value for the name).
Question:
Does a function like eval exist in NiFi where an attribute value containing Expression Language can be evaluated? We are using Expression Language to define and store dynamic file names in sql store, for instance using "enrollments_${now():format('yyyy')}_${now():format('MM')}.csv" to generate file name of "enrollments_<year>_<month>.csv". Though, it doesn't seem to work as filename attribute gets assigned this value in a literal manner; hence needing an extra step to evaluate this expression.
If this function doesn't exist, can this requirement be handled in a different way?
Any help will be highly appreciated.
Thanks!
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi