@Logann    NiFi does not offer local user creation for authentication.  There is no way to create local users and assign them passwords for the purpose of user authentication. User Authentication require either: 1. User certificates (always requested by NiFi during TLS handshake) 2. Spnego auth (Spnego auth challenge sent to browser if spnego properties configured in nifi.properties.  This request is only sent if 1 did not result in client certificate in response from client) 3. Configured login provider (uses login-provider configured in login-identity-providers.xml and referenced in nifi.properties file.  Only used if both 1 and 2 did not provide client/user authentication already). 4. NiFi will also support other OpenID Connect supported authentication providers. Hope this helps, Matt ... View more

@abhinav_joshi  This is something that only affects the Apache NiFi releases only. The removal of some nars is documented in the Apache NiFi migration guidance wiki here: https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance   "Starting in 1.10, the following nars were removed from the default convenience binary.  These include kite-nar, kafka-0-8-nar, flume-nar, media-nar, druid-controller-service-api-nar, druid-nar, other-graph-services-nar.  You can still get them from the various artifact repositories and use them in your flows but we cannot bundle them due to space limitations by default." For Apache NiFi users, I recommend that users create a custom NiFi lib directory by adding the following property to their nifi.properties file:   nifi.nar.library.directory.<unique-string>=/<path to>/custom-lib1 for exmaple: nifi.nar.library.directory.NiFi-1-10-nars=/nars/nifi-1-10-lib   You can add as many custom lib directories as you want.  Then place any nars that are noted as removed via the migration guidance documentation into one of these custom lib paths before upgrading.  Then as part of your upgrade process add the above property to your nifi.properties file. Then later if additional nar bundles are deprecated, you can create another custom lib dir or just add those nars to your existing custom lib directory before upgrading.  This allows you to avoid the downtime you encountered.   The Cloudera releases of HDF and CFM include are built with all the nars by default currently as they are not affected by space limitations. Hope this helps, Matt ... View more

@memad    If your GetFile processor is consuming files before they have finished writing there are a few changes that may help: 1. How are files being written in to the directory?  The default "File Filter" will ignore files that start with a ".".  If it is possible to change how files are being written to the directory, that will solve your issue through a file filter.  For example.... writing new files to directory as ".<filename>" and upon successful write does a rename to remove the dot (this is how ssh works).  But you can of course setup any file filter that works for you, 2. Assuming the process that is writing files to the directory is always updating the timestamp on the file, you can use the "Minimum File Age" property to prevent the GetFile from consuming a file until the last modified timestamp in the file has not updated for the configured amount of time.  This works in most cases, except when there may be long pauses in the write process that exceeds the configured Min File Age time. Hope this helps, Matt ... View more

@venkii    You need to login to your secured NiFi-Registry and make sure all your NiFi nodes have been authorized for both the following "Special Privileges": 1. "Read" for "Can Manage Buckets" 2. "Can proxy user requests"   Click on wrench icon in upper right corner to manage your users in NiFi-Registry. Then find your NiFi nodes in the list of USERS and click on the "manage user" pencil icon to the far right side.   Hope this helps, Matt ... View more

@Petr_Simik    No matter which processor you are looking at the stats presented all tell you the same information: In  <-- Tells you how many FlowFile were processed from one or more inbound connections over the last rolling 5 minute window.  With this processor you have it configured the "wait mode" to leave the FlowFile on the inbound connection.  So the processor is constantly looking at the file over and over again until the configured expiration time has elapsed. Read/Write. <-- Tells you how much FlowFile content was read from or written to the NiFi content repository (helps user identify processors that may be disk I/O heavy)   Out. <-- Tells you how many FlowFiles have been released to an outbound connection over the last rolling 5 minute window.  Here you see a number that reflects only those flowfiles that expired and where sent to your outbound expired connection. Tasks/Time. <-- Tells you how many threads this processor completed execution over the last rolling 5 minutes and the total cumulative time those threads consumed from the CPU.   (helps user identify what processors consume lots of CPU time) So the stats you are seeing are not surprising. While this processor works for your use case i guess, it has overhead needing to connect to a distributed map cache on every execution against an inbound FlowFile.  If your intent is only to delay a FlowFile for 1 second before it proceeds down the flow path, a better solution may be to just use an updateAttribute processor that creates an attribute with current time and RouteOnAttribute processor that checks to see if that recorded time plus 1000 ms is less than current time.  Then loop that check until it is not. Hope this helps, Matt ... View more

@venk  What you have run into at this point is a known issue. Your cluster was originally setup and running unsecured over HTTP port 8080. NiFi records the details of the nodes that are part of the cluster.  It does that so on later restarts it know that it should still be waiting on additional nodes to join before allowing users to make changes to the canvas. The downside to this is that when you switched to being secured over HTTPS on port 9091, the cluster now thinks you should have twice the number of nodes as there really are. But this is an easy fix.  Within your NiFi's conf directory you will find the file "state-management.xml". Inside that file you will find a section for NiFi's "local-provider" that will contain the directory where you can find your local state.  This path is normally the same on every node. Shutdown your NiFi and go to this directory on every node in your cluster and delete the contents within that state directory.  Restart your NiFi and it will only create new entries for your secured nodes. https://issues.apache.org/jira/browse/NIFI-7255 Hope this helps, Matt ... View more

@Faerballert  Perhaps you clone your flowfile before the mergeContent processors. So whichever relationship you are connecting to your current mergeContent, you drag a second connection containing that same relationship to a parallel notification flow. Down this parallel flow path you use a replaceText processor to replace the content with the value from the attribute you want to merge.  Then you use a mergeContent processor on this path to merge these files using a "," as your delimiter.   Then from this mergeContent you do you notification. You may also want to open an Apache Jira with your use case and desired improvement for the existing mergeContent.  The more details the better. Hope this helps, Matt ... View more

@Koffi    When a NiFi node attempts to connect to an existing NiFi cluster, there are three files that are checked to make sure they match exactly between the connecting node and the existing copies in the cluster.  Those files are: 1. flow.xml.gz 2. users.xml (will only exist if NiFi is secured over https) 3. authorizations.xml (not to be confused with the NiFi authorizers.xml file.  Will only exist if NiFi is secured over https) The output in the nifi-app.log of the node should explain exactly what the mismatch was the first time it tried to connect to the cluster.   Hope this helps, Matt     ... View more

@domR    i) Do List processors w/ timestamp tracking store state locally? --- If you are running a standalone NiFi and not a NiFi cluster, all state will be stored locally on disk. --- If clustered, this depends on the list processor and how it is configured.  The ListFile processor can be configured to store state locally or remotely depending on your use case.  For example a ListFile is added to a NiFi cluster and every node is listing from a local path not shared across all nodes, you would want each node to store the listFile state locally since it would be unique per node and other nodes have no access to the directory one each node.  If your listFile is listing against a mounted directory that is mounted to every node in the cluster, the listFile should be configured for remote and configured t run on primary node only. --- Other list based processors all store state locally ONLY when it is a standalone NiFi.  Clustered NiFi installs will trigger store to be stored in zookeeper. ii) Does this state survive NiFi restarts? --- Yes, local state is stored on disk in NiFI's local state directory.  Cluster/Remote state is store in zookeeper.  State configurable is handled by the state-management.xml configuration file.   iii) If running on primary node only, would this mean when another primary node is chosen, the List processor would list any files it hasn't tracked (and re-ingress a large backlog of files if still there)? --- When a primary node change occurs, the primary node only processors on the previous primary node are asked to stop executing and the same processors on the newly elected primary node are asked to start.  On the new node, that processor will retrieve that last known state stored in zookeeper for that component before executing.  There is a small chance for some limited data duplication.  When old elected primary node processors are asked to stop that does not kill active threads. If the processor is in the middle of execution and does not complete (update cluster state in ZK) before newly elected primary node pulls cluster state when it starts to execute, some files may be listed again by newly elected node, but it will not list from beginning. iv) What out-of-box solutions can help to get around the issue of non-persisted non-distributed listing, or do we need custom auditing triggering individual listings? ---  NiFi does persist state through node restarts.   Note; You can right click on a processor that stores state and select "view state" to see what has been stored. You can also right click on a processor and select "view usage" to open the embedded documentation for that component. The embedded documentation will contain a "State Management:" section that will tell you if the component stores state and if that state is stored locally or cluster (ZK).   Hope this helps, Matt ... View more