Hi Matt, Thank very much for your reply. I have defined the 1xxx port number higher than the 1024. Walking through the other information you shared. ... View more

Hi Matt, Thanks again for replying. So here goes explaining the current state. 1) This started out "accidently" as a standalone config (Someone else issued a Start Nifi command BEFORE I had configured this for a clustered environment.) I followed your instructions and removed the flow.xml.gz file and the templates directory from a previous posting, then set up the configuration for a clustered environment. 2) starting Nifi with clustering syntax defined in all of the configuration files, seemed to be good idea, until AFTER the fact that we found out that a) I should not have edited the Nifi properties files logged on as ROOT (per Dhruv) and b) there was a leftover process id that was running a newer version of NiFi. (Hortonworks had told us to not run the current Apache Project version of 0.6.1, but to instead install 0.6.0 from their software download site.) So we killed the leftover process running 0.6.1, I am using 0.6.0 and I am now editing the files logged on as the NIFI user id. 3) Because of all of the files being owned initially by Root, whenever possible, I have issued CHOWN command i.e. CHOWN nifi:nifi filename. At least on the node which I am issuing the Start Nifi command from. There is one node for the NCM, and 3 other slave nodes. 4) So what happened yesterday: issued a Start Nifi command, which didn't throw any errors, the UI came up, but it didn't show it was attached to a cluster. So I went to the log files,to try and figure out why it doesn't see the cluster, only found one error "WARN' level error message, which is the one I shared. Next will reply to your questions: 1. you have two different installation of NiFi. No 2. One installation of NiFi is setup and configured to be a non-secure (http) NCM. Only one, it is setup to be NON-secure 3. One instance of NiFi is setup and configured to be a non-secure (http) Node. Only one, it is setup to be NON-secure 4. The # cluster common properties (cluster manager and nodes must have same values) # section in the nifi.properties files on both NCM and Node(s) are configured identical 5. In that section on both nifi.cluster.protocol.is.secure=false is configured as false (Cannot be true if running http.) its set to false 6. The # cluster node properties (only configure for cluster nodes) # has been configured only on your node. - The following properties in the above node section are configured nifi.cluster.is.node=true set to true nifi.cluster.node.unicast.manager.address= defined - example below nifi.cluster.node.unicast.manager.protocol.port= defined - example below and the port matched what you configured in the next section in your NCM. 8. The # cluster manager properties (only configure for cluster manager) # section has been configured on your NCM only. - nifi.cluster.is.manager=true yes, set to true, only on NCM node Below are the highlights of what I have set, on the SLAVE node, when I have been starting Nifi from. I commented out actual values of Host name and showed port numbers that are unique by entering 1xxx, 2xxx # web properties # nifi.web.war.directory=./lib nifi.web.http.host=Axxxxx.xxxxxx.com nifi.web.http.port=8080 nifi.web.https.host= nifi.web.https.port= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 # cluster common properties (cluster manager and nodes must have same values) # nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.is.secure=false nifi.cluster.protocol.socket.timeout=30 sec nifi.cluster.protocol.connection.handshake.timeout=45 sec # if multicast is used, then nifi.cluster.protocol.multicast.xxx properties must be configured # nifi.cluster.protocol.use.multicast=false nifi.cluster.protocol.multicast.address= nifi.cluster.protocol.multicast.port= nifi.cluster.protocol.multicast.service.broadcast.delay=500 ms nifi.cluster.protocol.multicast.service.locator.attempts=3 nifi.cluster.protocol.multicast.service.locator.attempts.delay=1 sec # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=Axxxxxx.xxxxxx.com nifi.cluster.node.protocol.port=2xxxx nifi.cluster.node.protocol.threads=2 # if multicast is not used, nifi.cluster.node.unicast.xxx must have same values as nifi.cluster.manager.xxx # nifi.cluster.node.unicast.manager.address=Bxxxxx.xxxxxx.com nifi.cluster.node.unicast.manager.protocol.port=1xxx # cluster manager properties (only configure for cluster manager) # nifi.cluster.is.manager=false nifi.cluster.manager.address= nifi.cluster.manager.protocol.port= nifi.cluster.manager.node.firewall.file= nifi.cluster.manager.node.event.history.size=10 nifi.cluster.manager.node.api.connection.timeout=30 sec nifi.cluster.manager.node.api.read.timeout=30 sec nifi.cluster.manager.node.api.request.threads=10 nifi.cluster.manager.flow.retrieval.delay=5 sec nifi.cluster.manager.protocol.threads=10 nifi.cluster.manager.safemode.duration=0 sec ... View more

Hi to you both, Thank you both for replying. To reply to the first response: there are no firewalls between the nodes. Hi Matt: These are all configured for http only. I have not defined any port numbers anywhere in the configuration for HTTPS related parameters, due to the fact that I was told not to secure this configuration. Also, noticing now that flow.xml.gz seems to be empty on the Node where i am starting NiFi from... -rw-r--r-- 1 nifi nifi 0 Jun 2 15:30 flow.xml.gz in the past, this file looked like this: -rw-r--r-- 1 nifi nifi 1112 Jun 2 11:25 flow.xml.gz Could this be an factor as well? After issuing the start command, I can access the DataFlow UI, but it doesn't have any indicator that it sees a clustered environment. Thanks for any additional feedback. ... View more

@mclark Hi Matt, I really appreciate your replies. This is great idea for you to create: "How to setup my first non-secured NiFi cluster.". ... View more

Hi Matt, That was a beautiful explanation, and much appreciated, if I was going to secure the configuration. I was told not to secure this configuration. By reading your explanation for # 5: 5. You can now navigate to the non-secure (http) address for your NiFi which lets everyone in as anonymous with full access. ==> Go to the user management UI via the icon shown above and grant your user the "admin" role. Feedback: Is this is telling me, that even in an unsecured environment, I can go through the UI and grant myself the "Admin' role? If yes, then I am guessing at a minimum, I still need to do something with this entry in the file? <users> <userdn="[user dn - read only and admin]"> <rolename="ROLE_ADMIN"/> </user> </users> - I haven't had to do anything with editing an .xml file in a long time, so could the answer to this situation be, make the the following changes to this entry? <users> <userdn="[ADMIN - read only and admin]"> <rolename="ROLE_ADMIN"/> </user> </users> Things that could be changed or enhanced in the manual: 1) The manual assumes readers of the manual have editing experience in .xml files. Maybe the manual could say: if you are going to use the ADMIN user concept, edit this entry as follows: then supply the example.... Note: There is a great example to use when editing one the notification file: conf/bootstrap-notification-services.xml in the NiFi Admin guide. A complete example of configuring the Email service would look like the following: <service> <id>email-notification</id> <class>org.apache.nifi.bootstrap.notification.email.EmailNotificationService</class> <property name="SMTP Hostname">smtp.gmail.com</property> <property name="SMTP Port">587</property> <property name="SMTP Username">username@gmail.com</property> <property name="SMTP Password">super-secret-password</property> <property name="SMTP TLS">true</property> <property name="From">"NiFi Service Notifier"</property> <property name="To">username@gmail.com</property> </service> 2) Defining a few sentences in the manual to answer this scenario, would also be helpful: If the ADMIN user is actually associated with the concept of a 'Role' of privileges, do we need to activate the ADMIN user id in this file by removing the comment and supplying any other required information, even if this is NOT a secured configuration? Assuming that even in non secured configurations, some specific User will need to administer the NiFi configuration via the UI, versus allowing any user who can log in administer the tool? If that was an incorrect assumption on my part, please advise. Thank you. ... View more

Hello one last time, as follow-up to your last comment: "If the cluster NCM you are joining this node to already has an existing flow or templates, this node will fail to join because they will not match. NO need to reinstall to fix this if that is the case. Simply delete the flow.xml.gz file and the templates directory before starting it again. When it joins the cluster it will get the current flow and templates from the NCM." When the " start" command was originally issued (to accidently make this a standalone configuration) nothing had been configured - there was no NCM or node specific parameters defined. Bottomline: There were no other NiFi configurations previously in existence in this environment. Thanks again for your great answers and the articles you post to this site. They are very much appreciated. ... View more