Member since
02-14-2018
17
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
875 | 05-24-2019 12:00 AM |
05-24-2019
01:17 AM
Hi, Running NiFi 1.7.1 on Ubuntu 18.04. I have a nifi instance on a physical laptop on one site sending SCADA and local resource monitoring data via S2S to another nifi instance in AWS. Every day I have the laptop rebooting using cronjob. I have noticed that the queue size drops about 10MB when the reboot occurs. I thought nifi keeps the contents of the queue even when rebooted. Can I please ask for advice on how I can diagnose this, and to make sure there is no data being lost. I am trending the data on the AWS side and there doesn't seem to be any breaks... Below are trends of the resource monitoring on the laptop. The reboot occurs at 0400. Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
05-24-2019
12:00 AM
Worked it out - Ubuntu wired setting "Make Available to Other Users" was not set which was disabling the network when logged out.
... View more
01-22-2019
08:57 AM
Hi, I have NiFi 1.7.1 running as a service on Ubuntu 18.04. The service is running under root. The instance is set up to gather modbus data on a network using a custom processor, and send the data via S2S to another instance on a separate machine. The instance is also gathering other data such as nifi resources and sending them via S2S as well. My issue is that for some reason, when there is no user logged in to ubuntu, the flow of modbus data stops. Either via the ubuntu GUI or SSH. when I log back in, the flow starts back up. I cannot check if the service is running directly as I would need to log in for that, but the non modbus data continues to flow via S2S so I think the service is running. There doesn't seem to be anything obvious in the nifi-app.log, or bootstrap logs. Has anybody encountered unusual behaviour when the OS is not logged in? Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
09-27-2018
06:23 AM
Hi Akhil, It was helpful but I still do not have a solution for monitoring my NIFI instances. I will post a separate query. Thanks
... View more
09-25-2018
12:59 AM
Hi Akhil, Thanks for the prompt response. Is it possible to channel metrics from a standalone NiFi through an Ambari deployed Nifi to the Ambari dashboard? Thank you
... View more
09-21-2018
06:36 AM
Hi, I am keen to use Ambari to monitor my NiFi instances. I am running multiple standalone nifi instances on AWS EC2 machines. NiFi 1.5.0. However there doesn't seem to be any steps outlined to connect existing multiple nifi instances to a single Ambari. Could I please request some guidance on this? Thank you
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache NiFi
03-15-2018
12:35 AM
Hi Andy, Just letting you know that I got this working. Thank you for your help. What I did was to exchange the two nifi-cert.pem files between the two nifi instances and import them in the opposing truststores. At that point the RPG's status became "Forbidden". I then looked at the nifi-users.log file on the target Nifi, and I could see that the there was a user with DN (CN=<Nifi Instance 1 IP>, OU=NIFI) that was trying to authenticate. So I added that user in the target nifi, added to the global S2S policy and the individual port policies and voila! cheers
... View more
02-22-2018
06:47 AM
Hi Andy, thanks for your persistence. Made the change to the URL to look at port 8443. Still same warning about the trust anchors. I don't think I am importing the private key to the truststore of either instances. The .p12 file that the TLS Toolkit produces should be a certificate with the public key. It's the same file I import to a browser for it to access the Nifi UI. I don't understand why the nifi instances themselves have trouble authenticating. is the root public key the .pem file that is created by the toolkit? the .key file appears to be the private key. Thank you
... View more
02-18-2018
09:29 AM
Because the error message was referencing a previous instance IP, I have re-done the keystore and truststore on both instances using the TLS Toolkit. The commands I used on each instance were /data/download/nifi-toolkit-1.4.0/bin/tls-toolkit.sh standalone -n '<Nifi Instance 1 IP>' -C 'CN=Admin1,OU=DevOU' -O -o /data/security
/data/download/nifi-toolkit-1.4.0/bin/tls-toolkit.sh standalone -n '<Nifi Instance 2 IP>' -C 'CN=Admin2,OU=DevOU' -O -o /data/security
This produced the standard files on each instance, which I have incorporated to the conf folder. The .p12 files I imported to my browser, and I can securely access the UI of the instances as Admin1 or Admin2 after modifying the authorizers.xml and authorzed-users.xml. (port 8443). The S2S comms however is still giving me a headache. The S2S properties are as follows. The difference between the two instances is the IP number. I understand that you shouldn't need to have the IP number on the RPG instance side, but it seemed to work with the numbers on both sides when I had Nifi unsecured, so i have left them in. nifi.remote.input.host=<Nifi instance IP>
nifi.remote.input.secure=true
nifi.remote.input.socket.port=8899
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=8443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
I am using the 8899 port to have secure RAW protocol. On instance 1 the RPG I have set the URL to be https://<instance 2 IP>:8899/nifi/ with RAW protocol. I then imported the .p12 generated on each instance to the truststore of the other, using the keytool command keytool -v
-importkeystore -srckeystore CN=Admin1_OU=DevOU.p12 -srcstoretype PKCS12
-destkeystore truststore.jks -deststoretype JKS I now have an error on the RPG as follows: I'm pretty sure the issue is authentication. My thinking is that the truststore of instance 1 needs the private key of instance 2 imported and vice versa. I had thought if I imported the .p12 files this would do the trick but it has failed. I thought about the tutorial by Bryan and added users on both instances "CN=Admin1, OU=DevOU", "CN=Admin2, OU=DevOU" and even the one in the tutorial "CN=localhost, OU=NIFI" and gave each user policies for the site to site retrieval and added the users to the port policies too, but the error remains. I've tried different combination of RAW and HTTPS in the S2S properties but they don't seem to help. If I can get any further guidance, that would be appreciated. Thank you
... View more
02-15-2018
07:07 AM
Hi I have just had a go at the tutorial but there's a few aspects that is confusing me. The toolkit will generate nifi.property files for the two instances but their S2S properties are as follows: # Site to Site properties instance 1 nifi.remote.input.host=localhost nifi.remote.input.secure=true nifi.remote.input.socket.port=10443 nifi.remote.input.http.enabled=true ifi.remote.input.http.transaction.ttl=30 sec # Site to Site properties instance 2 nifi.remote.input.host=localhost nifi.remote.input.secure=true nifi.remote.input.socket.port=10444 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec Would I be correct in saying if I set up an RPG on instance 2 to point to Instance 1, I'd have to set it up with https://<Instance 1 IP>:10443/nifi ? How does the instance know which user to use, to access the ports etc? if I set up more users with less privileges, how would the instance know which one to use? I am guessing that because the nifi.remote.input.secure is set to true, it is not possible to secure a RAW protocol? Thank you
... View more
02-15-2018
06:15 AM
Hi Andy thanks for the response, I'll try to adapt the Bryan's tutorial to my application and report back. His tutorial seems to be for two NiFi instances running on one machine. My application is two NiFis running on separate cloud machines. Not sure if it makes a big difference. In the meantime here's the errors I get on my RPG. It's interesting that it seems to be referencing the 5... IP address. That was used on an EC2 instance that I created an image from, and I created this instance from that image. Thank you
... View more
02-14-2018
07:36 AM
Hi, I am trying to set up two NiFis (v1.4.0) on separate AWS EC2 instances and have them communicate securely. I have secured each NiFi so you need SSL authentication to access the UI on a browser. However I am not able to have the two instances communicate S2S. I have had success with UNSECURED NiFi instances with them communicating on RAW protocol. However I don't think it's secure so would like to lock down the S2S comms, as well as NiFi UI access as well. I'm pretty sure the properties nifi.remote.input.* and nifi.web.* are the ones to modify, but am confused as I can't seem find an example on the web for securing both the UI browser access and S2S comms. Could somebody please point me in the right direction? Thank you
... View more
Labels:
- Labels:
-
Apache NiFi