MiNiFi not connecting to Nifi

[#1]

Hi, im trying to set up a connection between MiNiFi running on a Pi to a NiFi server. But there doesnt seem to be a connection. The yaml file on MiNiFi is;

Flow Controller:
  name: listen-MiNiFi
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- name: GetFile
  class: org.apache.nifi.processors.standard.GetFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Batch Size: '10'
    File Filter: '[^\.].*'
    Ignore Hidden Files: 'true'
    Input Directory: /home/test
    Keep Source File: 'false'
    Maximum File Age:
    Maximum File Size:
    Minimum File Age: 0 sec
    Minimum File Size: 0 B
    Path Filter:
    Polling Interval: 0 sec
    Recurse Subdirectories: 'true'
Connections:
- name: GetFile/success/ab7ff1f6-0158-1000-0000-000011753a27
  source name: GetFile
  source relationship name: success
  destination name: ab7ff1f6-0158-1000-0000-000011753a27
  max work queue size: 10000
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- name: http://master.hadoop.com:9090/nifi
  url: http://master.hadoop.com:9090/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: ab7ff1f6-0158-1000-0000-000011753a27
    name: list-for-MiNiFi
    comment: ''
    max concurrent tasks: 1
    use compression: false
  - id: aaffbdc5-0158-1000-0000-00006f35ed0c
    name: listen-for-MiniFi
    comment: ''
    max concurrent tasks: 1
    use compression: false
Am i missing something?  have set the nifi remote input socket port to 10000
  

[#2]

Are there any errors in the minifi-app logs?

[#3]

not that i can see.

2016-12-07 16:31:23,639 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1 2016-12-07 16:31:37,641 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:31:37,928 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 285 milliseconds (Stop-the-world time = 110 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1 2016-12-07 16:31:37,928 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 286 milliseconds 2016-12-07 16:33:23,680 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1 2016-12-07 16:33:37,929 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:33:38,185 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 255 milliseconds (Stop-the-world time = 82 milliseconds, Clear Edit Logs time = 71 millis), max Transaction ID -1 2016-12-07 16:33:38,185 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 255 milliseconds 2016-12-07 16:35:23,721 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1 2016-12-07 16:35:38,186 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:35:38,481 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 294 milliseconds (Stop-the-world time = 103 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1 2016-12-07 16:35:38,481 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 295 milliseconds

[#4]

If you ping master.hadoop.com from the Raspberry Pi does it get a response?

[#5]

yes, and i can ssh into master too

[#6]

So what happens when you copy a file into /home/test? There should be something in the minifi-app.log when this happens.

[#7 · Accepted solution]

i started again and created another template using a generateflowfile and a remote processor group and still the same. below is a copy of the new yml file on MiNiFi

Flow Controller:
  name: flowfile
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- name: GenerateFlowFile
  class: org.apache.nifi.processors.standard.GenerateFlowFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Batch Size: '1'
    Data Format: Binary
    File Size: 100 MB
    Unique FlowFiles: 'false'
Connections:
- name: GenerateFlowFile/success/da4b4c07-0158-1000-0000-00004ed4df3c
  source name: GenerateFlowFile
  source relationship name: success
  destination name: da4b4c07-0158-1000-0000-00004ed4df3c
  max work queue size: 10000
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- name: http://192.168.1.99:9090/nifi
  url: http://192.168.1.99:9090/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: da4b4c07-0158-1000-0000-00004ed4df3c
    name: minifi
    comment: ''
    max concurrent tasks: 1
    use compression: false


Here are the last few lines from the log file after i run bin/minifi.sh start

 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:13:54,677 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:13:54,971 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 293 milliseconds (Stop-the-world time = 108 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1
2016-12-07 18:13:54,972 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 294 milliseconds
2016-12-07 18:15:25,997 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:15:54,973 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:15:55,326 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 352 milliseconds (Stop-the-world time = 167 milliseconds, Clear Edit Logs time = 73 millis), max Transaction ID -1
2016-12-07 18:15:55,327 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 353 milliseconds

[#8]

hi @Bryan Bende i restarted the minifi agent and all is working now, thanks for your time.

[#9]

ok glad it is working now!

[#10]

Hello! @Bryan Bende

I am also facing the same issue. I restarted minifi agent. Still the same problem. I am getting following error in ExecuteProcess Processor of MiNifi flow:-

problem.png

problem2.png

My NiFi and MiNifi are not connecting I guess. Your help will be of great value.

Thanks.