question Re: Using MiNiFi: Remote Process Group Connection Refused in Archives of Support Questions (Read Only)

Using MiNiFi: Remote Process Group Connection Refused

elloyd — Tue, 04 Apr 2017 23:00:35 GMT

We set up our minifi & flow according to this tutorial: https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

I have done it several times to verify it was done correctly.

Within the server that minifi is installed on, upon starting it, after initialization, in the minifi-app.log, we get this error: p.p1

2017-04-04 08:59:27,307 ERROR [Timer-Driven Process Thread-1] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://foo.xyz.abc.com:8080/nifi] failed to communicate with http://foo.xyz.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

We expected this was a port issue so we checked the ports using nmap -sT -O localhost on our Nifi Server and it seems the 10000 port is open. We also did a telnet on URL of our Nifi Server on our minifi server and it seemed to connect okay.

As a note: even when I just have a GenerateFlowFile -> RPG going to http://foo.xyz.abc.com:8080/nifi on my Nifi interface (essentially its remoting to itself), it still give me this same error as above.

Any guidance would be much appreciated.

Re: Using MiNiFi: Remote Process Group Connection Refused

bbende — Tue, 04 Apr 2017 23:32:54 GMT

I assume that when you access the NiFi UI in your browser you are going to http://foo.xyz.cequintecid.com:8080/nifi and that works fine right?

Can you perform the GenerateFlowFile -> RPG to self test and get the full stacktrace from nifi-app.log when the error happens. Thanks.

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Wed, 05 Apr 2017 00:20:27 GMT

Answer to your first question is yes.

Oddly enough for the second one, the nifi-app.log's last entry is from March 2nd. Im pretty baffled as to why this isn't logging this in there. It has to be logging it somewhere. I did a find in the CLI and there is only one nifi-app.log on the Nifi server. Although it does seem to be updating the nifi-user.log

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Wed, 05 Apr 2017 00:25:48 GMT

Also oddly enough, I've noticed that my nifi-app.log is owned by root instead of hdfs while everything else is owned by hdfs. Maybe that is why it isn't logging. Further down the rabbit hole

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Wed, 05 Apr 2017 01:31:03 GMT

Okay I changed the owner to HDFS and it started logging. Here are the results. It seems to be giving the same Connection Refused error as on the minifi log althought I also noticed this one:

"2017-04-04 18:28:17,894 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with 192.168.90.15:10000 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused"

2017-04-04 18:27:51,742 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.expiration.FileRemovalAction@7cbf9849 on Provenance Event file ./provenance_repository/28828679.prov.gz in 432121 nanos

2017-04-04 18:27:56,057 ERROR [Timer-Driven Process Thread-2] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

2017-04-04 18:28:06,159 ERROR [Timer-Driven Process Thread-7] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

2017-04-04 18:28:15,489 INFO [Site-to-Site Worker Thread-15] o.a.nifi.remote.SocketRemoteSiteListener Received connection from 192.168.90.15/192.168.90.15, User DN: null

2017-04-04 18:28:15,601 INFO [Site-to-Site Worker Thread-15] o.a.nifi.remote.SocketRemoteSiteListener Successfully negotiated ServerProtocol SocketFlowFileProtocol Version 6 with Peer[url=nifi://192.168.90.15:43335]

2017-04-04 18:28:15,713 INFO [Site-to-Site Worker Thread-15] o.a.n.r.p.s.SocketFlowFileServerProtocol Sending list of 1 peers back to client Peer[url=nifi://192.168.90.15:43335]

2017-04-04 18:28:16,160 INFO [Timer-Driven Process Thread-3] o.apache.nifi.remote.client.PeerSelector New Weighted Distribution of Nodes:

PeerStatus[hostname=192.168.90.15,port=10000,secure=false,flowFileCount=0] will receive 100.0% of data

2017-04-04 18:28:16,261 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

2017-04-04 18:28:17,369 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 29052330

2017-04-04 18:28:17,392 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (11 records) into single Provenance Log File ./provenance_repository/29052319.prov in 41 milliseconds

2017-04-04 18:28:17,394 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 52 records. In the past 5 minutes, 116 events have been written to the Provenance Repository, totaling 59.19 KB

2017-04-04 18:28:17,894 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with 192.168.90.15:10000 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused

2017-04-04 18:28:17,894 INFO [Site-to-Site Worker Thread-16] o.a.nifi.remote.SocketRemoteSiteListener Received connection from xyz.foo.abc.com/10.10.2.116, User DN: null

2017-04-04 18:28:17,905 INFO [Site-to-Site Worker Thread-16] o.a.nifi.remote.SocketRemoteSiteListener Successfully negotiated ServerProtocol SocketFlowFileProtocol Version 6 with Peer[url=nifi://xyz.foo.abc.com:46241]

2017-04-04 18:28:17,915 INFO [Site-to-Site Worker Thread-16] o.a.n.r.p.s.SocketFlowFileServerProtocol Sending list of 1 peers back to client Peer[url=nifi://xyz.foo.abc.com:46241]

2017-04-04 18:28:17,925 INFO [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@1a703270 Successfully refreshed Peer Status; remote instance consists of 1 peers

2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.UpdateMinimumEventId Updated Minimum Event ID for Provenance Event Repository - Minimum Event ID now 28837444

2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.lucene.UpdateMinimumEventId@63086b79 on Provenance Event file ./provenance_repository/28837433.prov.gz in 689201 nanos

2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Event file ./provenance_repository/28837433.prov.gz

2017-04-04 18:28:22,091 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Table-of-Contents file ./provenance_repository/toc/28837433.toc

2017-04-04 18:28:22,091 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.expiration.FileRemovalAction@7cbf9849 on Provenance Event file ./provenance_repository/28837433.prov.gz in 210909 nanos

2017-04-04 18:28:26,363 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused

Re: Using MiNiFi: Remote Process Group Connection Refused

bbende — Wed, 05 Apr 2017 04:09:52 GMT

Ok, are you doing RAW site-to-site or HTTP-based (this is an option in the RPG)?

And what do these properties look like in your nifi.properties?

# Site to Site properties

nifi.remote.input.host=

nifi.remote.input.secure=false

nifi.remote.input.socket.port=

nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Thu, 06 Apr 2017 00:46:16 GMT

We are using RAW and this is our nifi.properties file

# Site to Site properties

#nifi.remote.input.host=192.168.x.x

#nifi.remote.input.secure=false

#nifi.remote.input.socket.port=10000

#nifi.remote.input.http.enabled=true

#nifi.remote.input.http.transaction.ttl=30 sec

We've actually switched to a Nifi to Nifi using RPG now wondering if Minifi was the problem but it still delivers the exactly same Connection Refused message.

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Thu, 06 Apr 2017 21:39:07 GMT

Not sure if it helps but we also found this error message that appeared before the Connection Refused message:

2017-04-06 07:14:05,610 ERROR [Timer-Driven Process Thread-6] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=From MiNiFi,targets=http://foo.xyz.abc.com:8080/nifi] failed to communicate with http://foo.xyz.abc.com:8080/nifi due to org.apache.nifi.remote.exception.HandshakeException: org.apache.nifi.remote.exception.ProtocolException: Expected to receive ResponseCode, but the stream did not have a ResponseCode

Re: Using MiNiFi: Remote Process Group Connection Refused

bbende — Fri, 07 Apr 2017 03:18:46 GMT

Nothing is really jumping out at me, it feels like something isn't configured correctly, but I'm not sure what.

Have you tried making nifi.remote.input.host the same value as the hostname you are using for the UI?

Just wondering since that was a 192 address and the web host looks like a hostname. The web host and s2s host can be different, but just wondering if there is some issue using the 192 address in this case.

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Sat, 08 Apr 2017 01:24:10 GMT

I am rather unsure what you mean.

nifi.remote.input.host is set to the same value as the hostname we are using for the Nifi UI, I think.

nifi.remote.input.host= 192.168.x.x

Hostname for UI= 192.168.x.x:8080/nifi

Do you mean to add the port and /nifi onto it in the nifi.properties folder?

Re: Using MiNiFi: Remote Process Group Connection Refused

bbende — Sat, 08 Apr 2017 03:19:07 GMT

Sorry I may have just mis-understood the information...

In all of the logs it showed foo.xyz.abc.com and then for nifi.remote.input.host you said it was 192.168.x.x. I realize both are obfuscated, but I was assuming these were different values, otherwise I thought you would have put foo.xyz.abc.com for both of them.

If nifi.web.http.host and nifi.remote.input.host are the same in your nifi.properties, then you can ignore my previous comment.

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Mon, 10 Apr 2017 22:41:18 GMT

So I think perhaps theres some confusion.

foo.xyz.abc.com is the url of the location of the receiving server and of course, 8080/nifi is added to access the nifi interface

192.168.x.x is the IP address of the sending server. They are different values and should be.

In response to your comment: If nifi.web.http.host and nifi.remote.input.host are the same in your nifi.properties, then you can ignore my previous comment.

They are not the same - in fact, nifi.web.http.host is blank and when I put the same value as nifi.remote,input.host, the receiving servers Nifi UI does not work anymore (as expected)

It is interesting to note that I was actually able to get Nifi to talk to itself through localhost using these values

nifi.remote.input.host=localhost

nifi.remote.input.secure=false

nifi.remote.input.socket.port=10000

nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

I think my next step is to install Nifi on another server than my sending server and try to send to my receiving server to try to figure out which server is the problem child.

Re: Using MiNiFi: Remote Process Group Connection Refused

bbende — Mon, 10 Apr 2017 22:50:21 GMT

Ok nifi.remote.input.host should not be the value of a remote server, it should either be blank just like the web host which means it will bind to all interfaces, or it should be a specific hostname/ip of the current node that you are editing nifi.properties on.

Here is how it works, lets say you have a standalone NiFi instance with a RPG trying to send data to a 3 node cluster... In the RPG you put the URL of the target cluster's UI like http://somehost:8080/nifi, the RPG then makes a REST call to http://somehost:8080/nifi to ask it for the information about all the nodes in the cluster, the response contains the value of nifi.remote.input.host and nifi.remote.input.socket.port for each of the three nodes in the destination cluster, so now the source instance knows where to send data two.

Re: Using MiNiFi: Remote Process Group Connection Refused

elloyd — Tue, 11 Apr 2017 02:06:45 GMT

Hey Bryan

That was it! If you check out the tutorial at https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

the wording makes it seem to me like we were meant to put in the IP address for the sending server. Thank you so much for your help. A week and half to figure this out.

Re: Using MiNiFi: Remote Process Group Connection Refused

TimothySpann — Wed, 03 Jan 2018 00:22:36 GMT

Switch to HTTP, make sure everything has permissions. check firewalls. I have had wrong input names sometimes in the config.yml check those.

Re: Using MiNiFi: Remote Process Group Connection Refused

Nabil — Fri, 27 Mar 2020 07:17:17 GMT

u can see this artical: https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP%28S%29+as+a+transport+mechanism+for+Site-to-Site

Re: Using MiNiFi: Remote Process Group Connection Refused

Nabil — Fri, 27 Mar 2020 07:19:14 GMT

you have to fiil in nifi.remote.input.host= at conf/nifi.properties. this work when you deploy nifi to actual host