Using MiNiFi: Remote Process Group Connection Refused

[#1]

We set up our minifi & flow according to this tutorial: https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

I have done it several times to verify it was done correctly.

Within the server that minifi is installed on, upon starting it, after initialization, in the minifi-app.log, we get this error: p.p1

2017-04-04 08:59:27,307 ERROR [Timer-Driven Process Thread-1] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://foo.xyz.abc.com:8080/nifi] failed to communicate with http://foo.xyz.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

We expected this was a port issue so we checked the ports using nmap -sT -O localhost on our Nifi Server and it seems the 10000 port is open. We also did a telnet on URL of our Nifi Server on our minifi server and it seemed to connect okay.

As a note: even when I just have a GenerateFlowFile -> RPG going to http://foo.xyz.abc.com:8080/nifi on my Nifi interface (essentially its remoting to itself), it still give me this same error as above.

Any guidance would be much appreciated.

[#2 · Accepted solution]

I assume that when you access the NiFi UI in your browser you are going to http://foo.xyz.cequintecid.com:8080/nifi and that works fine right?

Can you perform the GenerateFlowFile -> RPG to self test and get the full stacktrace from nifi-app.log when the error happens. Thanks.

[#3]

Answer to your first question is yes.

Oddly enough for the second one, the nifi-app.log's last entry is from March 2nd. Im pretty baffled as to why this isn't logging this in there. It has to be logging it somewhere. I did a find in the CLI and there is only one nifi-app.log on the Nifi server. Although it does seem to be updating the nifi-user.log

[#4]

Also oddly enough, I've noticed that my nifi-app.log is owned by root instead of hdfs while everything else is owned by hdfs. Maybe that is why it isn't logging. Further down the rabbit hole

[#5]

Okay I changed the owner to HDFS and it started logging. Here are the results. It seems to be giving the same Connection Refused error as on the minifi log althought I also noticed this one:

"2017-04-04 18:28:17,894 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with 192.168.90.15:10000 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused"

2017-04-04 18:27:51,742 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.expiration.FileRemovalAction@7cbf9849 on Provenance Event file ./provenance_repository/28828679.prov.gz in 432121 nanos

2017-04-04 18:27:56,057 ERROR [Timer-Driven Process Thread-2] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

2017-04-04 18:28:06,159 ERROR [Timer-Driven Process Thread-7] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

2017-04-04 18:28:15,489 INFO [Site-to-Site Worker Thread-15] o.a.nifi.remote.SocketRemoteSiteListener Received connection from 192.168.90.15/192.168.90.15, User DN: null

2017-04-04 18:28:15,601 INFO [Site-to-Site Worker Thread-15] o.a.nifi.remote.SocketRemoteSiteListener Successfully negotiated ServerProtocol SocketFlowFileProtocol Version 6 with Peer[url=nifi://192.168.90.15:43335]

2017-04-04 18:28:15,713 INFO [Site-to-Site Worker Thread-15] o.a.n.r.p.s.SocketFlowFileServerProtocol Sending list of 1 peers back to client Peer[url=nifi://192.168.90.15:43335]

2017-04-04 18:28:16,160 INFO [Timer-Driven Process Thread-3] o.apache.nifi.remote.client.PeerSelector New Weighted Distribution of Nodes:

PeerStatus[hostname=192.168.90.15,port=10000,secure=false,flowFileCount=0] will receive 100.0% of data

2017-04-04 18:28:16,261 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

2017-04-04 18:28:17,369 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 29052330

2017-04-04 18:28:17,392 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (11 records) into single Provenance Log File ./provenance_repository/29052319.prov in 41 milliseconds

2017-04-04 18:28:17,394 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 52 records. In the past 5 minutes, 116 events have been written to the Provenance Repository, totaling 59.19 KB

2017-04-04 18:28:17,894 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with 192.168.90.15:10000 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused

2017-04-04 18:28:17,894 INFO [Site-to-Site Worker Thread-16] o.a.nifi.remote.SocketRemoteSiteListener Received connection from xyz.foo.abc.com/10.10.2.116, User DN: null

2017-04-04 18:28:17,905 INFO [Site-to-Site Worker Thread-16] o.a.nifi.remote.SocketRemoteSiteListener Successfully negotiated ServerProtocol SocketFlowFileProtocol Version 6 with Peer[url=nifi://xyz.foo.abc.com:46241]

2017-04-04 18:28:17,915 INFO [Site-to-Site Worker Thread-16] o.a.n.r.p.s.SocketFlowFileServerProtocol Sending list of 1 peers back to client Peer[url=nifi://xyz.foo.abc.com:46241]

2017-04-04 18:28:17,925 INFO [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@1a703270 Successfully refreshed Peer Status; remote instance consists of 1 peers

2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.UpdateMinimumEventId Updated Minimum Event ID for Provenance Event Repository - Minimum Event ID now 28837444

2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.lucene.UpdateMinimumEventId@63086b79 on Provenance Event file ./provenance_repository/28837433.prov.gz in 689201 nanos

2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Event file ./provenance_repository/28837433.prov.gz

2017-04-04 18:28:22,091 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Table-of-Contents file ./provenance_repository/toc/28837433.toc

2017-04-04 18:28:22,091 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.expiration.FileRemovalAction@7cbf9849 on Provenance Event file ./provenance_repository/28837433.prov.gz in 210909 nanos

2017-04-04 18:28:26,363 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

[#6]

 u can see this artical: https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP%28S%29+as+a+transport+mechanism+for+S...

[#7]

you have to fiil in nifi.remote.input.host= at conf/nifi.properties. this work when you deploy nifi to actual host 

[#8]

Ok, are you doing RAW site-to-site or HTTP-based (this is an option in the RPG)?

And what do these properties look like in your nifi.properties?

# Site to Site properties

nifi.remote.input.host=

nifi.remote.input.secure=false

nifi.remote.input.socket.port=

nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

[#9]

We are using RAW and this is our nifi.properties file

# Site to Site properties

#nifi.remote.input.host=192.168.x.x

#nifi.remote.input.secure=false

#nifi.remote.input.socket.port=10000

#nifi.remote.input.http.enabled=true

#nifi.remote.input.http.transaction.ttl=30 sec

We've actually switched to a Nifi to Nifi using RPG now wondering if Minifi was the problem but it still delivers the exactly same Connection Refused message.

[#10]

Not sure if it helps but we also found this error message that appeared before the Connection Refused message:

2017-04-06 07:14:05,610 ERROR [Timer-Driven Process Thread-6] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=From MiNiFi,targets=http://foo.xyz.abc.com:8080/nifi] failed to communicate with http://foo.xyz.abc.com:8080/nifi due to org.apache.nifi.remote.exception.HandshakeException: org.apache.nifi.remote.exception.ProtocolException: Expected to receive ResponseCode, but the stream did not have a ResponseCode