Glad I could help and good to hear you are now up and running. ... View more

You can edit files as root. Editing files does not change ownership. You just need to make sure at the end of editing all files are owned by the user who will be running yoUR NiFi instances. Give yourself a fresh start and delete the flow.tar on your NCM and flow.xml.gz and templates dir on your Node. So at the end of configuring your two NiFi installs (one install configured to be NCM and one separate install configured to be a Node), you started your NCM successfully? Looking in the nifi-app.log for your NCM, do you see the following lines: 2016-06-03 ... INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs: 2016-06-03 ... INFO [main] org.apache.nifi.web.server.JettyServer https://Bxxxxx.xxxxxx.com:8080/nifi You then go to your other NiFi installation configured as your Node and start it. After it has started successfully it will start attempting to send heartbeats to Bxxxxx.xxxxxxx.com on port 1xxx. You should see these incoming heartbeats logged in the nifi-app.log on your NCM. Do you see these? INFO [Process NCM Request-1] o.a.n.c.p.impl.SocketProtocolListener Received request 411684b2-25cb-461f-978e-fb3bda6a7ef0 from Axxxxx.xxxxxx.com INFO [Process NCM Request-1] o.a.n.c.manager.impl.WebClusterManager Node Event: (......) 'Connection requested from new node. Setting status to connecting.' After that the NCM will either mark the node as connected or given a reason for not allowing it to connect If you are not seeing these heartbeats in the NCM nifi-app.log, then something is blocking the TCP traffic on the specified port. I did notice in the above example you provided 1xxx as your cluster manger port. Is that port above 1024? Ports <= 1024 are reserved and can't be used by non root users. If you are running your NCM as a user other then root (as it sounds by the above) NiFi will fail to bind to that port for listening for these heartbeats. Matt ... View more

A fresh install of NiFi has no flow.xml.gz file until after it is started for the first time. Are these fresh NiFi installs or installations that were previously run standalone? - if that is the case you can't simply tell them they are nodes and NCMs and expect it to work. Your NCM does not run with a flow.xml.gz like your nodes and standalone instances do. The NCM uses a flow.tar file. The flow.tar would be created on startup and contain an empty flow.xml. When you started your Node (with existing flow.xml.gz file) it would have communicated with NCM but been rejected because the flow on the node would not have matched what was on the NCM. If you are looking to migrate form a standalone instance to a cluster, I would suggest reading this: https://community.hortonworks.com/content/kbentry/9203/how-to-migrate-a-standalone-nifi-into-a-nifi-clust.html Let me make sure understand your environment: 1. you have two different installation of NiFi. 2. One installation of NiFi is setup and configured to be a non-secure (http) NCM 3. One instance of NiFi is setup and configured to be a non-secure (http) Node. 4. The # cluster common properties (cluster manager and nodes must have same values) # section in the nifi.properties files on both NCM and Node(s) are configured identical 5. In that section on both nifi.cluster.protocol.is.secure=false is configured as false (Cannot be true if running http.) 6. The # cluster node properties (only configure for cluster nodes) # has been configured only on your node. - The following properties in the above node section are configured nifi.cluster.is.node=true nifi.cluster.node.unicast.manager.address= nifi.cluster.node.unicast.manager.protocol.port= and the port matched what you configured in the next section in your NCM. 8. The # cluster manager properties (only configure for cluster manager) # section has been configured on your NCM only. - nifi.cluster.is.manager=true Thanks, Matt ... View more

Are these https or http configured cluster NCM and Node(s)? NCM needs to be able to communicate with the http(s) port and node.protocol port configured in the nifi .properties file on the Node(s). Node needs to be able to communicate with the cluster manager protocol port configured in the nifi.properties file on the NCM. Thanks, Matt ... View more

Ahmad, The line you are seeing in the nifi-bootstrap.log indicates the JVM started successfully. You need to check the nifi-app.log to make sure the application loaded successfully. In the nifi-app.log you will find the following lines if the application successfully loaded: 2016-05-31 10:46:44,347 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs: 2016-05-31 10:46:44,347 INFO [main] org.apache.nifi.web.server.JettyServer http://<someaddress or FQDN>:8088/nifi Verify that the hostname or IP displayed on this line is reachable/resolvable on the system you are running your web browser from. Thanks, Matt ... View more

Thank you for your feedback. I will look into creating an article to cover this topic. Perhaps "How to setup my first non-secured NiFi cluster." ... View more

PJ, Roles only work with a secured NiFi. The intent of step 5 was to give a backdoor method for properly populating the first needed entry in the authorized-users.xml file. You can either follow step 5 (backdoor method for creating first needed Admin user for https setup) or you can get your DN from your cert or ldap to manually populate that authorized-users.xml. Bottom line is if you are being told you cannot secure your NiFi and to leave it running unsecure with http, you cannot create user roles. User roles are only used by a secured https configured NiFi. If someone is telling you that you can setup user roles within a http configured NiFi, they are unfortunately misinformed. An example entry is below should you decide to secure your NiFi so you can make use of this feature in the future: <users> <userdn="CN=John Doe, OU=MyBusiness, O=MyOrg, L=Baltimore, ST=MD, C=US"> <rolename="ROLE_ADMIN"/> </user> </users> Keep in mind that in order for this to work, the userdn above has to match exactly how it is recorded in the users certificate or within ldap for user John Doe in this example. What this does is authorize the authenticated user "John Doe" with the "Admin" role. A user can be assigned multiple roles as well. An example of how that would look is as follows: <users> <userdn="CN=John Doe, OU=MyBusiness, O=MyOrg, L=Baltimore, ST=MD, C=US"> <rolename="ROLE_ADMIN"/> <rolename="ROLE_DFM"/> <rolename="ROLE_PROVENANCE"/> </user> </users> In that example, the authenticated user "John Doe" has been authorized with "Admin", "Dataflow Manager", and "Provenance" user roles. Hope this adds clarity to my previous response. Thanks, Matt ... View more

Hello, Pierre is correct about what the swap threshold is used for. For speed and efficiency, NiFi hold all the FlowFile attributes associated with each FlowFile in JVM memory. In case where queue develop this can result in considerable memory usage. So NiFi has established a default swapping threshold of 20,000 FlowFiles per connection. What this means is that once a queue reaches 30,000 FlowFiles, 10,000 will be swapped out to disk. The 20,000 that are the next to be worked on based on the connection prioritization scheme are left in memory. NiFi will continue to swap 10,000 FlowFiles at a time to disk as the queue continues to grow. Keep in mind that files swapped out must be swapped back in before they can be worked on by the destination processor. NiFi does not throw away any data unless expiration has been set on connections. As long as you have sufficient disk space to hold the data, it will continue to queue. I suggest reading through this article if you have not already: ( https://community.hortonworks.com/content/kbentry/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html ) That being said, there are dangers to allowing your disk to fill to 100% capacity, so as Pierre mentioned you should be setting backpressure throughout your dataflow to trigger upstream processor to stop and eventually stopping pulling in new data. Thanks, Matt ... View more

Hello, In order to secure access to your NiFi instance/cluster, NiFi must be configured to run securely via https. Section 6 of the guide linked above correctly states that your NiFi must be configured to run securely (HTTPS) and have an authentication mechanism (user certificates. ldap, or kerberos) in place. Without a secure setup, all users who access the NiFi UI get in with anonymous access which gives all of them full access so all aspects of the NiFi UI. The DN is what uniquely identifies each user and/or server that accesses your NiFi. If using ssl certificates as your authentication mechanism, the DN will be found inside the certificate and would have been generated during the certificates creation phase. There is an article here ( https://community.hortonworks.com/articles/17293/how-to-create-user-generated-keys-for-securing-nif.html ) that walks you through creating your own keystores and truststores for securing your NiFi (https). It also covers creating user certificates if that is the authentication mechanism you choose to use. You can also use ldap or kerberos as your authentication mechanism, but you will need to setup or use an existing ldap or kerberos infastructure. Users with the "admin" role have the ability to authenticate in to the secured NiFi UI. From there they can access the user management interface via this icon . This interface will allow users that have the "admin" role to approve the access of other users who have requested through the secured NiFi UI. The reason you need to manually add the first "admin" user is because otherwise you have no users who can access the UI to approve requests. If you are unsure how to extract your user DN from your configure authentication mechanism, you can do the following: 1. You still need to setup your NiFi securely. You can use the procedure linked above to create the needed keystores and truststores to do so. 2. Configure your nifi.properties file for secure (https) and non-secure (http) access. You will need to use unique ports for each. (8080 for http and 8443 for https for example). 3. Navigate to the https address for your NiFi instance. If you are using user certificates, you will need to have followed that section of the above linked article to create your user key and load it in to your browser. If setup for ldap, provide your ldap username and password when the NiFI UI prompts you. 4. You will then be prompted to request access if you successfully authenticated. This is the authorization request portion. after requesting access the screen will say pending approval. 5. You can now navigate to the non-secure (http) address for your NiFi which lets everyone in as anonymous with full access. Go to the user management UI via the icon shown above and grant your user the "admin" role. You can now go back to the secured NiFi UI address and gain controlled access. Don't forget to go back in your nifi.properties file and remove the http configure to prevent uncontrolled annoymous access at this point. There are other roles that authorize authenticated users to do different things within the UI: 1. adminstrator: Can add or remove authorized roles for other users. Can purge flow configuration change history. 2. Data Flow Manager: Can build, manipulate, modify, start, stop, and/or delete dataflows on the NiFi canvas. 3. Read Only: Can access UI and view the configuration of items on the canvas, but cannot build, manipulate, modify, start, stop, and/or delete any of it. 4. Provenance: Users with this added role can search any stored provenance data. I am not sure where the confusion came from with regards to setting up controlled access via http, but if you can point me in the right direction I will do my best to get the documentation updated so it is more clear. Thank you, Matt ... View more