@Gubbi The next solution here is to just add each route for today, yesterday, day before yesterday.   Then route all 3 to the next proc.    Anything not matching won't be routed.  ... View more

@Gubbi   The solution here is a now minute 24 hours:   Yesterday:   ${now():minus(86400000):format('MM-dd-yyyy hh:mm:ss') }   Day Before Yesterday:   ${now():minus(86400000):minus(86400000):format('MM-dd-yyyy hh:mm:ss') } ... View more

@saaga119 The solution here is to create the external table in the format you need to query the raw xml data.  Seems like you have this done already.   Next create a native hive table.  It could still be external too. The new table should be the schema for the fields you want with required ORC lines.  You can also choose some other text format; for example: csv.   Last INSERT INTO new_table SELECT * FROM external_table.   The main idea here is to use external and staging tables, then INSERT INTO SELECT to fill the final table you want.  Final table could then be optimized for performance (ORC).   This kind of idea also allows multiple external/raw/staging tables able to be combined (select w/ join) into a single final optimized table (orc, compression, partitions, buckets, etc). ... View more

@Savi    I think you only need one date format in your nifi expression language.     Next, it seems you know what format it is upstream (source), and what it needs to be for downstream.   Based on your expression need,  you should play the flow up to the csv reader processor and then List Queue on the FlowFiles and inspect the attributes.  Adjust the expression, and retest until the attribute format meets the required format.    Inspecting the dates upstream won't show you what the date looks like after expression and before the reader.   Often times I find myself testing a flow step by step, each step confirming the attributes needed are correct for the next step.  Never moving forward to the next step until I am sure the attributes are correct after visual inspection. ... View more