Hello @Rohit Ravishankar ListFile uses last modified timestamp of files. It tracks the latest last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which ListFile already picked, then the file won't be picked with ListFile logic. There is an existing JIRA to discuss about the behavior [1]. Min/Max filter is used to filter-out files that is too-young (min) or too-old (max) files. Even if a file passed these condition if its last modified timestamp is older than the latest on already listed, it won't be picked. If your use-case requires processing input files in 'descending last modified timestamp ' order, then I'd recommend using GetFile (keepSourceFile = false) and PriorityAttributePrioritizer [2] combination. [1] https://issues.apache.org/jira/browse/NIFI-2383 [2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization I hope this helps. ... View more

Hello @Gabriel Queiroz I'm surprised to know that there's no existing processor that removes a key from distributed map cache. Would you submit a JIRA issue to request that functionality if possible? In the mean while, if you encounter such shortcomings, you can address it by writing a custom processor with ExecuteScript or InvokeScriptedProcessor in most cases. Those processors let you write custom processor using your favorite scripting engine. I've written an example, using Groovy to remove a key from distributed map cache. It will work with your use-case I think. https://gist.github.com/ijokarumawak/14d560fec5a052b3a157b38a11955772 ... View more

Hello @Gabriel Queiroz I hope I understood your use-case properly. How about using EvaluateJsonPath to extract each values into attributes so that ExecuteSQL can refer? A template file for above example flow is available here: https://gist.github.com/ijokarumawak/6d2bbfe48809e036008153f15113ab80 ... View more

Hello @Ilya Li NiFi stores states as binary representation starting with encoding version. That's why you see invisible characters I believe, and copy-paste will not work [1]. Theoretically, but zk-migrator.sh would be able to copy zookeeper state which is available in NiFi Toolkit [2]. I did a simple experimentation using zk-migrator.sh. I was able to copy state from a processor to another by following steps: 1. Export state from Zk to a JSON file: ./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk.json -r 2. Copy exported zk.json to a different file, and edit: Changed connectString and servers to 'localhost:2181' as the tool doesn't allow loading data to the same destination. Also removed other paths. So that the file only contains two objects, one containing 'connectString' and the other containing exported state that you want to copy. Change the path with target processor UUID, so that it will be loaded into the target processor: [ { "connectString": "localhost:2182", "servers": [ "localhost:2182" ], "path": "/" }, { "path": "/nifi/components/05f810b2-015d-1000-ffff-ffff80b2e488", "data": [1, 0, ...], "stat": { "czxid": 552, "mzxid": 553, "ctime": 1499042915091, "mtime": 1499042915154, "version": 1, "cversion": 0, "aversion": 0, "ephemeralOwner": 0, "dataLength": 79, "numChildren": 0, "pzxid": 552 }, "acls": [ { "perms": 31, "id": { "scheme": "world", "id": "anyone" } } ], "ephemeralOwner": 0 } ] 3. Load modified zk json: ./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk-copy.dat -s This way, you can copy state from a processor to another. It may work if those processors use the same structure of state (same keys ... etc). Alternative way is to write a simple Zk client app referring ZookeeperStateProvider's serialize and deserialize methods [1]. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/providers/zookeeper/ZooKeeperStateProvider.java#L254 [2] https://nifi.apache.org/download.html Hope this helps, Koji ... View more