question Re: MiNiFi not connecting to Nifi in Support Questions

MiNiFi not connecting to Nifi

r_young — Wed, 07 Dec 2016 23:38:59 GMT

Hi, im trying to set up a connection between MiNiFi running on a Pi to a NiFi server. But there doesnt seem to be a connection. The yaml file on MiNiFi is;

Flow Controller:
  name: listen-MiNiFi
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- name: GetFile
  class: org.apache.nifi.processors.standard.GetFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Batch Size: '10'
    File Filter: '[^\.].*'
    Ignore Hidden Files: 'true'
    Input Directory: /home/test
    Keep Source File: 'false'
    Maximum File Age:
    Maximum File Size:
    Minimum File Age: 0 sec
    Minimum File Size: 0 B
    Path Filter:
    Polling Interval: 0 sec
    Recurse Subdirectories: 'true'
Connections:
- name: GetFile/success/ab7ff1f6-0158-1000-0000-000011753a27
  source name: GetFile
  source relationship name: success
  destination name: ab7ff1f6-0158-1000-0000-000011753a27
  max work queue size: 10000
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- name: http://master.hadoop.com:9090/nifi
  url: http://master.hadoop.com:9090/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: ab7ff1f6-0158-1000-0000-000011753a27
    name: list-for-MiNiFi
    comment: ''
    max concurrent tasks: 1
    use compression: false
  - id: aaffbdc5-0158-1000-0000-00006f35ed0c
    name: listen-for-MiniFi
    comment: ''
    max concurrent tasks: 1
    use compression: false
Am i missing something?  have set the nifi remote input socket port to 10000

Re: MiNiFi not connecting to Nifi

jpercivall — Thu, 08 Dec 2016 00:32:42 GMT

Are there any errors in the minifi-app logs?

Re: MiNiFi not connecting to Nifi

r_young — Thu, 08 Dec 2016 00:37:49 GMT

not that i can see.

2016-12-07 16:31:23,639 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1 2016-12-07 16:31:37,641 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:31:37,928 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 285 milliseconds (Stop-the-world time = 110 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1 2016-12-07 16:31:37,928 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 286 milliseconds 2016-12-07 16:33:23,680 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1 2016-12-07 16:33:37,929 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:33:38,185 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 255 milliseconds (Stop-the-world time = 82 milliseconds, Clear Edit Logs time = 71 millis), max Transaction ID -1 2016-12-07 16:33:38,185 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 255 milliseconds 2016-12-07 16:35:23,721 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1 2016-12-07 16:35:38,186 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:35:38,481 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 294 milliseconds (Stop-the-world time = 103 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1 2016-12-07 16:35:38,481 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 295 milliseconds

Re: MiNiFi not connecting to Nifi

bbende — Thu, 08 Dec 2016 01:37:05 GMT

If you ping master.hadoop.com from the Raspberry Pi does it get a response?

Re: MiNiFi not connecting to Nifi

r_young — Thu, 08 Dec 2016 01:40:05 GMT

yes, and i can ssh into master too

Re: MiNiFi not connecting to Nifi

bbende — Thu, 08 Dec 2016 02:04:07 GMT

So what happens when you copy a file into /home/test? There should be something in the minifi-app.log when this happens.

Re: MiNiFi not connecting to Nifi

r_young — Thu, 08 Dec 2016 02:17:42 GMT

i started again and created another template using a generateflowfile and a remote processor group and still the same. below is a copy of the new yml file on MiNiFi

Flow Controller:
  name: flowfile
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- name: GenerateFlowFile
  class: org.apache.nifi.processors.standard.GenerateFlowFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Batch Size: '1'
    Data Format: Binary
    File Size: 100 MB
    Unique FlowFiles: 'false'
Connections:
- name: GenerateFlowFile/success/da4b4c07-0158-1000-0000-00004ed4df3c
  source name: GenerateFlowFile
  source relationship name: success
  destination name: da4b4c07-0158-1000-0000-00004ed4df3c
  max work queue size: 10000
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- name: http://192.168.1.99:9090/nifi
  url: http://192.168.1.99:9090/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: da4b4c07-0158-1000-0000-00004ed4df3c
    name: minifi
    comment: ''
    max concurrent tasks: 1
    use compression: false


Here are the last few lines from the log file after i run bin/minifi.sh start

 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:13:54,677 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:13:54,971 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 293 milliseconds (Stop-the-world time = 108 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1
2016-12-07 18:13:54,972 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 294 milliseconds
2016-12-07 18:15:25,997 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:15:54,973 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:15:55,326 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 352 milliseconds (Stop-the-world time = 167 milliseconds, Clear Edit Logs time = 73 millis), max Transaction ID -1
2016-12-07 18:15:55,327 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 353 milliseconds

Re: MiNiFi not connecting to Nifi

r_young — Thu, 08 Dec 2016 02:47:26 GMT

hi @Bryan Bende i restarted the minifi agent and all is working now, thanks for your time.

Re: MiNiFi not connecting to Nifi

bbende — Thu, 08 Dec 2016 03:43:28 GMT

ok glad it is working now!

Re: MiNiFi not connecting to Nifi

divyasodha10 — Wed, 07 Mar 2018 19:24:25 GMT

Hello! @Bryan Bende

I am also facing the same issue. I restarted minifi agent. Still the same problem. I am getting following error in ExecuteProcess Processor of MiNifi flow:-

problem.png

problem2.png

My NiFi and MiNifi are not connecting I guess. Your help will be of great value.

Thanks.

Re: MiNiFi not connecting to Nifi

TimothySpann — Wed, 07 Mar 2018 20:57:59 GMT

Make sure you have permissions and you can run that file independently.

Make sure you are running that flow converted to config.yml on the RPI.

I like to wrap my Python in a shell script like

python3 /home/pi/WeatherStation.py

make sure MInIFi user has permissions.

Re: MiNiFi not connecting to Nifi

divyasodha10 — Fri, 09 Mar 2018 16:55:19 GMT

@Timothy Spann

How to check permissions and verify whether flow is running on Rpi?

I am seeing this kind of logs in minifi-app.log:-

"2018-03-09 13:00:01,454 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with 192.168.2.3:9999 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused 2018-03-09 13:00:01,475 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@cf5b60 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster 2018-03-09 13:00:01,742 INFO [Remote Process Group 96072a53-f650-44b7-9987-2a99e1c4f3c2: http://192.168.2.3:8090/nifi Thread-1] org.apache.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://192.168.2.3:8090: The target server failed to respond 2018-03-09 13:00:01,744 INFO [Remote Process Group 96072a53-f650-44b7-9987-2a99e1c4f3c2: http://192.168.2.3:8090/nifi Thread-1] org.apache.http.impl.execchain.RetryExec Retrying request to {}->http://192.168.2.3:8090 2018-03-09 13:00:01,747 INFO [Timer-Driven Process Thread-2] org.apache.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://192.168.2.3:8090: The target server failed to respond 2018-03-09 13:00:01,747 INFO [Timer-Driven Process Thread-2] org.apache.http.impl.execchain.RetryExec Retrying request to {}->http://192.168.2.3:8090"

Re: MiNiFi not connecting to Nifi

TimothySpann — Fri, 09 Mar 2018 20:37:54 GMT

There you go, MiniFi is not able to connect to NiFi .

ConnectException: Connection refuse

is this the correct server 192.168.2.3

is that the correct IP and port?

do you have NiFi setup to receive remote?

what is the url you access NiFi with?

https://community.hortonworks.com/articles/88473/site-to-site-communication-between-secured-https-a.html

With NiFi 1.5 you can only have one domain or IP to point to.

Make sure you follow the setup directions here:

https://nifi.apache.org/minifi/getting-started.html

Configuring NiFi to Receive Data

First, check that $NIFI_HOME/conf/nifi.properties has a Site to Site input socket port specified:

# Site to Site properties
nifi.remote.input.host=
nifi.remote.input.secure=false
nifi.remote.input.socket.port=1026
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec

Restart NiFi so changes take effect.

https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

Re: MiNiFi not connecting to Nifi

divyasodha10 — Sun, 08 Jul 2018 13:41:50 GMT

Even with the correcr server i.e. 10.0.75.1, I am getting the same error. What's the problem?

Re: MiNiFi not connecting to Nifi

divyasodha10 — Sun, 08 Jul 2018 13:46:04 GMT

I am not able to ping from Raspberry Pi to Hortanworks Server from laptop but vice-versa is possible.

Re: MiNiFi not connecting to Nifi

TimothySpann — Mon, 09 Jul 2018 04:14:51 GMT

if you can't ping, then networking is setup wrong on RPI. Home networking often has issues and blocks to stop this. You must make sure you have open ports and no firewalls blocking. Also make sure you really know the correct IP for your laptop/server. Then edit the /etc/hosts on the RPI put in that ip next to a name for it. Make sure NiFi is ready to accept remotes. then reboot. try ping again.

Re: MiNiFi not connecting to Nifi

divyasodha10 — Mon, 09 Jul 2018 14:28:33 GMT

Which ports should be open ? And how to make sure NiFi is ready to accept remotes?