@Dale1979    It would appear you have a multi-node NiFi cluster that is using the NiFi CA to issue the certificate for each NiFi node.   Within a NiFi cluster, one node will be elected cluster coordinator. While you can access your NiFi cluster from any node in the cluster, any user requests (including displaying the UI) need to be replicated to the cluster coordinator.  This replication request is performed by the node which you are authenticated in to on behalf of you as the user.  Thus requiring that the node itself has been authorized to act as a proxy for your user.   Within your authorizers.xml file you should find a file-user-group-provider which should have been configured a separate "Initial User Identity <num>" property for each of your NiFi nodes. The file-user-group-provider is used add the initial users in the users.xml file. The value would match the DN of the certificate found in each node's keystore.jks file.  Using you output as an example, you should see a line like this:   <property name="Initial User Identity 2">CN=server.domain:port, OU=NIFI</property>   Above would be followed by additional similar lines with new number for each of your other NiFi nodes.   Note: Having a port number in your CN name is unexpected.   IMPORTANT:  NiFi will only generate the users.xml and authorizations.xml files from the configuration in the authorizers.xml file the very first time.  If you make edits to either the file-user-group-provider (build users.xml) or File-access-provider (builds authorizations.xml), those changes will not be reflected in the already existing users.xml and authorizations.xml files.  You must delete these files so they are recreated.  The expectation is that following successful securing of NiFi, all additional users and authorizers are added directly via the NiFi UI.   Hope this helps, Matt ... View more

@jspuri    I suggest making the following configuration changes:   1. The zoo.cfg files on both your ZK nodes should be the same tickTime=2000 initLimit=5 syncLimit=2 dataDir=/home/ec2-user/zookeeper clientPort=2181 server.1=ec2-server-1.compute-1.amazonaws.com:2888:3888 server.2=ec2-server-2.compute-1.amazonaws.com:2888:3888  Note: For proper Quorum zookeeper cluster should also have an odd number of servers (3 or 5)   2. In NiFi state-management.xml <property name="Connect String">ec2-server-1.compute-1.amazonaws.com:2888:3888,ec2-server-2.compute-1.amazonaws.com:2888:3888</property>   3. in nifi.properties file nifi.zookeeper.connect.string=ec2-server-1.compute-1.amazonaws.com:2888:3888,ec2-server-2.compute-1.amazonaws.com:2888:3888     Hope this helps, Matt ... View more

@Kit2020    You will want to use the MergeContent or MergeRecord processor to merge you incoming data from Kafka. NiFi FlowFiles consists of two parts: 1. FlowFile attributes/metadata   --> This information is held in heap memory 2. FlowFile content . --> held in NiFi's content repository on disk.   The merge based processor above will hold the FlowFile attributes of every FlowFile being merged (allocated to a merge bin) in heap memory.  Considering you are talking about merging 10 million FlowFiles, you will not be able to merge all of these using a single merge processor without likely encountering an out of memory condition.  A better approach is to have two merge processor in series with the first merging batches of min 10000 to 20000 and the second merging those into another batch resulting in the merging of all FlowFiles.      A merge processor bin is eligible to be merged when the minimum set values are met.  Meaning if you set min entires/records to 1 and max to 10000, the bin can merge with only one FlowFile. At time of execution the thread grabs what is in inbound connection at that moment in time and allocates it to a bin. Then checks if that bin met mins and if so merges it.  So makes sure your minimums  on your multiple merge processors are set high enough (for example 10000 on first and 1000 on merge processor number 2, result 10,000,000 merged FlowFiles)   Now if you do not know the exact number of records you need to merge, set the second merge processor mins to a higher value than you expect to receive.  Then set your "Max bin age" to a value you can accept for data latency.  The max bin age is your force merge setting.  So even if min vales are not reached, a bin that has existed for this length of time will be forced to merge.  Setting bin age on both processor is important.  With high value on second then first.  This allows time for a bin from merge one (typically last created) that may not meet 10000 min to get merged while second is still waiting.   Hope this helps, Matt ... View more

@Tine    The JettyWebSocketClient NiFi controller does support NiFi EL (variable registry only)  in the WebSocketURI property.  This means that you can means you could use for example ${token} in the URL you set in this Controller service.  Variables can be set on the process group.    So you have two option here and both involve using the NiFi Rest-API. Both start with obtaining the new token (perhaps via a rest-api call using the invokeHTTP processor?   1. Use NiFi's rest-API to stop, modify, and start theJettyWebSocketClient 2. Use NiFi's rest-API to modify a NiFi variable set on a process group.  Advantage here is the token variable can be used by multiple processors with the process group (or sub-process groups) where it was defined.  Allowing you to make only one update.   So consider building a flow that upon failure you trigger an invokeHTTP to get a new token and then another invokeHTTP to update the variable via NiFi's rest_API.  Then loop failed FlowFile back to processor that failed to try again. When working with NiFi's rest-api, it is helps to use your browser's developer tools to capture the rest api calls as you manually perform them via the NiFi UI.  NiFi's rest-api documentation is also available within the embedded docs of your NiFi installation (global menu --> Help).   Hope this helps, Matt ... View more

@koshay    This error is telling you that the user running the NiFi process is unable to see that file. ...java.nio.file.NoSuchFileException If the file really does exist at "/cygdrive/Windows/System32/LogFiles/Firewall/pfirewall.log", then the issue is more likely permissions on one of the directories in that path preventing the NiFi java process user from being able traverse that directory tree to the log file.    If the NiFi Java process user was able to traverse that entire directory tree, we would expect a different error related to permissions on the log file.   Hope this helps, Matt     ... View more

@girish6    Normally a flow like this would always be running.  The processors being feed by the input port in PG2 are not going to do any work until they receive that upstream FlowFile that is coming from PG1.  Processors are designed to yield when there is not work to do to avoid excessive CPU usage. Do you have concerns with how this works?   Thanks, Matt ... View more

@girish6    At a high level, anything you can do via the NiFi UI can also be done via rest-api calls.  So while no processor exists specifically for the purpose of triggering another component to enable and start, you could use the invokeHTTP processor(s) to make the necessary rest-api calls to start other components.   Getting the rest-api calls needed can be done using developer tools available in most browsers.  You can capture the rest api calls as you perform the actions within the NiFi UI.  Most developer tools allow you to save as curl for example which gives you the exact rest-api endpoint called, all the headers, and other relevant input for the rest-api endpoint (often json). NiFi dataflow architecture is typically designed as an always on setup. You have not shared much about your use case as to why you need to start PG2 only when PG1 has a specific outcome.  Maybe there are alternate design solutions here.   Hope this helps, Matt ... View more