I am rather unsure what you mean. nifi.remote.input.host is set to the same value as the hostname we are using for the Nifi UI, I think. nifi.remote.input.host= 192.168.x.x Hostname for UI= 192.168.x.x:8080/nifi Do you mean to add the port and /nifi onto it in the nifi.properties folder? ... View more

Not sure if it helps but we also found this error message that appeared before the Connection Refused message: 2017-04-06 07:14:05,610 ERROR [Timer-Driven Process Thread-6] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=From MiNiFi,targets=http://foo.xyz.abc.com:8080/nifi] failed to communicate with http://foo.xyz.abc.com:8080/nifi due to org.apache.nifi.remote.exception.HandshakeException: org.apache.nifi.remote.exception.ProtocolException: Expected to receive ResponseCode, but the stream did not have a ResponseCode ... View more

We are using RAW and this is our nifi.properties file # Site to Site properties #nifi.remote.input.host=192.168.x.x #nifi.remote.input.secure=false #nifi.remote.input.socket.port=10000 #nifi.remote.input.http.enabled=true #nifi.remote.input.http.transaction.ttl=30 sec We've actually switched to a Nifi to Nifi using RPG now wondering if Minifi was the problem but it still delivers the exactly same Connection Refused message. ... View more

Okay I changed the owner to HDFS and it started logging. Here are the results. It seems to be giving the same Connection Refused error as on the minifi log althought I also noticed this one: "2017-04-04 18:28:17,894 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with 192.168.90.15:10000 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused" 2017-04-04 18:27:51,742 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.expiration.FileRemovalAction@7cbf9849 on Provenance Event file ./provenance_repository/28828679.prov.gz in 432121 nanos 2017-04-04 18:27:56,057 ERROR [Timer-Driven Process Thread-2] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused 2017-04-04 18:28:06,159 ERROR [Timer-Driven Process Thread-7] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused 2017-04-04 18:28:15,489 INFO [Site-to-Site Worker Thread-15] o.a.nifi.remote.SocketRemoteSiteListener Received connection from 192.168.90.15/192.168.90.15, User DN: null 2017-04-04 18:28:15,601 INFO [Site-to-Site Worker Thread-15] o.a.nifi.remote.SocketRemoteSiteListener Successfully negotiated ServerProtocol SocketFlowFileProtocol Version 6 with Peer[url=nifi://192.168.90.15:43335] 2017-04-04 18:28:15,713 INFO [Site-to-Site Worker Thread-15] o.a.n.r.p.s.SocketFlowFileServerProtocol Sending list of 1 peers back to client Peer[url=nifi://192.168.90.15:43335] 2017-04-04 18:28:16,160 INFO [Timer-Driven Process Thread-3] o.apache.nifi.remote.client.PeerSelector New Weighted Distribution of Nodes: PeerStatus[hostname=192.168.90.15,port=10000,secure=false,flowFileCount=0] will receive 100.0% of data 2017-04-04 18:28:16,261 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused 2017-04-04 18:28:17,369 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 29052330 2017-04-04 18:28:17,392 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (11 records) into single Provenance Log File ./provenance_repository/29052319.prov in 41 milliseconds 2017-04-04 18:28:17,394 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 52 records. In the past 5 minutes, 116 events have been written to the Provenance Repository, totaling 59.19 KB 2017-04-04 18:28:17,894 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with 192.168.90.15:10000 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused 2017-04-04 18:28:17,894 INFO [Site-to-Site Worker Thread-16] o.a.nifi.remote.SocketRemoteSiteListener Received connection from xyz.foo.abc.com/10.10.2.116, User DN: null 2017-04-04 18:28:17,905 INFO [Site-to-Site Worker Thread-16] o.a.nifi.remote.SocketRemoteSiteListener Successfully negotiated ServerProtocol SocketFlowFileProtocol Version 6 with Peer[url=nifi://xyz.foo.abc.com:46241] 2017-04-04 18:28:17,915 INFO [Site-to-Site Worker Thread-16] o.a.n.r.p.s.SocketFlowFileServerProtocol Sending list of 1 peers back to client Peer[url=nifi://xyz.foo.abc.com:46241] 2017-04-04 18:28:17,925 INFO [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@1a703270 Successfully refreshed Peer Status; remote instance consists of 1 peers 2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.UpdateMinimumEventId Updated Minimum Event ID for Provenance Event Repository - Minimum Event ID now 28837444 2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.lucene.UpdateMinimumEventId@63086b79 on Provenance Event file ./provenance_repository/28837433.prov.gz in 689201 nanos 2017-04-04 18:28:22,090 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Event file ./provenance_repository/28837433.prov.gz 2017-04-04 18:28:22,091 INFO [Provenance Maintenance Thread-3] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Table-of-Contents file ./provenance_repository/toc/28837433.toc 2017-04-04 18:28:22,091 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.expiration.FileRemovalAction@7cbf9849 on Provenance Event file ./provenance_repository/28837433.prov.gz in 210909 nanos 2017-04-04 18:28:26,363 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=MinifiSource,target=http://xyz.foo.abc.com:8080/nifi] failed to communicate with http://xyz.foo.abc.com:8080/nifi due to java.net.ConnectException: Connection refused ... View more

Also oddly enough, I've noticed that my nifi-app.log is owned by root instead of hdfs while everything else is owned by hdfs. Maybe that is why it isn't logging. Further down the rabbit hole ... View more

Answer to your first question is yes. Oddly enough for the second one, the nifi-app.log's last entry is from March 2nd. Im pretty baffled as to why this isn't logging this in there. It has to be logging it somewhere. I did a find in the CLI and there is only one nifi-app.log on the Nifi server. Although it does seem to be updating the nifi-user.log ... View more

Thanks Bryan. Im really not sure why this data that came from Syslog into Kafka is not matching the Syslog RFC standards... We didn't alter the data in Kafka at all. This is an example of what an event looks like... looks like standard Syslog data to me. Im hesitant about using ExtractText but will if I have to. Mar 21 18:10:37 xxxfw01 /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.10.1.000 from 00:a0:ad:0e:4f:a1 to 00:e1:ed:34:26:c5 If I was to extract the Timestamp from the file using ExtractText and say assign it to Timestamp as an attribute, then can I use that new Timestamp attribute to create the filename and directory instead of using the time it was consumed from Kafka (which is a 3 min delay from the timestamps in the log files) ... View more

Unfortunately (and I'm not sure why), our Syslog results are not conforming to those standards cause we get the error shown in the screenshot. Im not sure if this has something to do with the Syslog data going into Kafka and then us having to Consume it from Kafka, maybe its altering the format. Is there a way to get the timestamp data from data from a Kafka topic? ... View more