Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

MiNiFi not connecting to Nifi

avatar
Expert Contributor

Hi, im trying to set up a connection between MiNiFi running on a Pi to a NiFi server. But there doesnt seem to be a connection. The yaml file on MiNiFi is;

Flow Controller:
  name: listen-MiNiFi
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- name: GetFile
  class: org.apache.nifi.processors.standard.GetFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Batch Size: '10'
    File Filter: '[^\.].*'
    Ignore Hidden Files: 'true'
    Input Directory: /home/test
    Keep Source File: 'false'
    Maximum File Age:
    Maximum File Size:
    Minimum File Age: 0 sec
    Minimum File Size: 0 B
    Path Filter:
    Polling Interval: 0 sec
    Recurse Subdirectories: 'true'
Connections:
- name: GetFile/success/ab7ff1f6-0158-1000-0000-000011753a27
  source name: GetFile
  source relationship name: success
  destination name: ab7ff1f6-0158-1000-0000-000011753a27
  max work queue size: 10000
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- name: http://master.hadoop.com:9090/nifi
  url: http://master.hadoop.com:9090/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: ab7ff1f6-0158-1000-0000-000011753a27
    name: list-for-MiNiFi
    comment: ''
    max concurrent tasks: 1
    use compression: false
  - id: aaffbdc5-0158-1000-0000-00006f35ed0c
    name: listen-for-MiniFi
    comment: ''
    max concurrent tasks: 1
    use compression: false
Am i missing something?  have set the nifi remote input socket port to 10000
  
1 ACCEPTED SOLUTION

avatar
Expert Contributor

i started again and created another template using a generateflowfile and a remote processor group and still the same. below is a copy of the new yml file on MiNiFi

Flow Controller:
  name: flowfile
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- name: GenerateFlowFile
  class: org.apache.nifi.processors.standard.GenerateFlowFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Batch Size: '1'
    Data Format: Binary
    File Size: 100 MB
    Unique FlowFiles: 'false'
Connections:
- name: GenerateFlowFile/success/da4b4c07-0158-1000-0000-00004ed4df3c
  source name: GenerateFlowFile
  source relationship name: success
  destination name: da4b4c07-0158-1000-0000-00004ed4df3c
  max work queue size: 10000
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- name: http://192.168.1.99:9090/nifi
  url: http://192.168.1.99:9090/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: da4b4c07-0158-1000-0000-00004ed4df3c
    name: minifi
    comment: ''
    max concurrent tasks: 1
    use compression: false


Here are the last few lines from the log file after i run bin/minifi.sh start

 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:13:54,677 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:13:54,971 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 293 milliseconds (Stop-the-world time = 108 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1
2016-12-07 18:13:54,972 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 294 milliseconds
2016-12-07 18:15:25,997 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:15:54,973 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:15:55,326 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 352 milliseconds (Stop-the-world time = 167 milliseconds, Clear Edit Logs time = 73 millis), max Transaction ID -1
2016-12-07 18:15:55,327 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 353 milliseconds


View solution in original post

16 REPLIES 16

avatar

Are there any errors in the minifi-app logs?

avatar
Expert Contributor

not that i can see.

2016-12-07 16:31:23,639 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1 2016-12-07 16:31:37,641 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:31:37,928 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 285 milliseconds (Stop-the-world time = 110 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1 2016-12-07 16:31:37,928 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 286 milliseconds 2016-12-07 16:33:23,680 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1 2016-12-07 16:33:37,929 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:33:38,185 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 255 milliseconds (Stop-the-world time = 82 milliseconds, Clear Edit Logs time = 71 millis), max Transaction ID -1 2016-12-07 16:33:38,185 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 255 milliseconds 2016-12-07 16:35:23,721 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1 2016-12-07 16:35:38,186 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2016-12-07 16:35:38,481 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 294 milliseconds (Stop-the-world time = 103 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1 2016-12-07 16:35:38,481 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 295 milliseconds

avatar
Master Guru

If you ping master.hadoop.com from the Raspberry Pi does it get a response?

avatar
Expert Contributor

yes, and i can ssh into master too

avatar
Master Guru

So what happens when you copy a file into /home/test? There should be something in the minifi-app.log when this happens.

avatar
Expert Contributor

i started again and created another template using a generateflowfile and a remote processor group and still the same. below is a copy of the new yml file on MiNiFi

Flow Controller:
  name: flowfile
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- name: GenerateFlowFile
  class: org.apache.nifi.processors.standard.GenerateFlowFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Batch Size: '1'
    Data Format: Binary
    File Size: 100 MB
    Unique FlowFiles: 'false'
Connections:
- name: GenerateFlowFile/success/da4b4c07-0158-1000-0000-00004ed4df3c
  source name: GenerateFlowFile
  source relationship name: success
  destination name: da4b4c07-0158-1000-0000-00004ed4df3c
  max work queue size: 10000
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- name: http://192.168.1.99:9090/nifi
  url: http://192.168.1.99:9090/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: da4b4c07-0158-1000-0000-00004ed4df3c
    name: minifi
    comment: ''
    max concurrent tasks: 1
    use compression: false


Here are the last few lines from the log file after i run bin/minifi.sh start

 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:13:54,677 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:13:54,971 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 293 milliseconds (Stop-the-world time = 108 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1
2016-12-07 18:13:54,972 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 294 milliseconds
2016-12-07 18:15:25,997 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:15:54,973 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:15:55,326 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 352 milliseconds (Stop-the-world time = 167 milliseconds, Clear Edit Logs time = 73 millis), max Transaction ID -1
2016-12-07 18:15:55,327 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 353 milliseconds


avatar
Expert Contributor

hi @Bryan Bende i restarted the minifi agent and all is working now, thanks for your time.

avatar
Master Guru

ok glad it is working now!

avatar

Hello! @Bryan Bende

I am also facing the same issue. I restarted minifi agent. Still the same problem. I am getting following error in ExecuteProcess Processor of MiNifi flow:-

problem.png

problem2.png

My NiFi and MiNifi are not connecting I guess. Your help will be of great value.

Thanks.