Member since
10-27-2016
122
Posts
13
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1540 | 01-08-2017 07:14 PM | |
6255 | 12-07-2016 06:17 PM | |
4504 | 11-08-2016 08:49 PM |
12-07-2016
06:47 PM
hi @Bryan Bende i restarted the minifi agent and all is working now, thanks for your time.
... View more
12-07-2016
06:17 PM
1 Kudo
i started again and created another template using a generateflowfile and a remote processor group and still the same. below is a copy of the new yml file on MiNiFi Flow Controller:
name: flowfile
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- name: GenerateFlowFile
class: org.apache.nifi.processors.standard.GenerateFlowFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list: []
Properties:
Batch Size: '1'
Data Format: Binary
File Size: 100 MB
Unique FlowFiles: 'false'
Connections:
- name: GenerateFlowFile/success/da4b4c07-0158-1000-0000-00004ed4df3c
source name: GenerateFlowFile
source relationship name: success
destination name: da4b4c07-0158-1000-0000-00004ed4df3c
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: ''
Remote Processing Groups:
- name: http://192.168.1.99:9090/nifi
url: http://192.168.1.99:9090/nifi
comment: ''
timeout: 30 sec
yield period: 10 sec
Input Ports:
- id: da4b4c07-0158-1000-0000-00004ed4df3c
name: minifi
comment: ''
max concurrent tasks: 1
use compression: false
Here are the last few lines from the log file after i run bin/minifi.sh start
0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:13:54,677 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:13:54,971 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 293 milliseconds (Stop-the-world time = 108 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1
2016-12-07 18:13:54,972 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 294 milliseconds
2016-12-07 18:15:25,997 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 39 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 18:15:54,973 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 18:15:55,326 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 352 milliseconds (Stop-the-world time = 167 milliseconds, Clear Edit Logs time = 73 millis), max Transaction ID -1
2016-12-07 18:15:55,327 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 353 milliseconds
... View more
12-07-2016
04:37 PM
not that i can see. 2016-12-07 16:31:23,639 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1
2016-12-07 16:31:37,641 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 16:31:37,928 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 285 milliseconds (Stop-the-world time = 110 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1
2016-12-07 16:31:37,928 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 286 milliseconds
2016-12-07 16:33:23,680 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 9 millis), max Transaction ID -1
2016-12-07 16:33:37,929 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 16:33:38,185 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 255 milliseconds (Stop-the-world time = 82 milliseconds, Clear Edit Logs time = 71 millis), max Transaction ID -1
2016-12-07 16:33:38,185 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 255 milliseconds
2016-12-07 16:35:23,721 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1ec2968 checkpointed with 0 Records and 0 Swap Files in 40 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 8 millis), max Transaction ID -1
2016-12-07 16:35:38,186 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-12-07 16:35:38,481 INFO [pool-72-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@11d0bbc checkpointed with 0 Records and 0 Swap Files in 294 milliseconds (Stop-the-world time = 103 milliseconds, Clear Edit Logs time = 72 millis), max Transaction ID -1
2016-12-07 16:35:38,481 INFO [pool-72-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 295 milliseconds
... View more
12-07-2016
03:38 PM
1 Kudo
Hi, im trying to set up a connection between MiNiFi running on a Pi to a NiFi server. But there doesnt seem to be a connection. The yaml file on MiNiFi is; Flow Controller:
name: listen-MiNiFi
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- name: GetFile
class: org.apache.nifi.processors.standard.GetFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list: []
Properties:
Batch Size: '10'
File Filter: '[^\.].*'
Ignore Hidden Files: 'true'
Input Directory: /home/test
Keep Source File: 'false'
Maximum File Age:
Maximum File Size:
Minimum File Age: 0 sec
Minimum File Size: 0 B
Path Filter:
Polling Interval: 0 sec
Recurse Subdirectories: 'true'
Connections:
- name: GetFile/success/ab7ff1f6-0158-1000-0000-000011753a27
source name: GetFile
source relationship name: success
destination name: ab7ff1f6-0158-1000-0000-000011753a27
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: ''
Remote Processing Groups:
- name: http://master.hadoop.com:9090/nifi
url: http://master.hadoop.com:9090/nifi
comment: ''
timeout: 30 sec
yield period: 10 sec
Input Ports:
- id: ab7ff1f6-0158-1000-0000-000011753a27
name: list-for-MiNiFi
comment: ''
max concurrent tasks: 1
use compression: false
- id: aaffbdc5-0158-1000-0000-00006f35ed0c
name: listen-for-MiniFi
comment: ''
max concurrent tasks: 1
use compression: false
Am i missing something? have set the nifi remote input socket port to 10000
... View more
Labels:
- Labels:
-
Apache MiNiFi
11-28-2016
06:52 PM
Hi @Andy LoPresto, thank you for the info. i have just read the link on Stack overflow. it is an excellent description and very informative.
... View more
11-28-2016
04:52 PM
I think if you designate more RAM to the guest than the host it may cause it to crash. try it with 4GB of RAM
... View more
11-28-2016
09:28 AM
Ok thank you, i was guessing thats what MiNiFi did. Thanks for your answers, very helpful
... View more
11-27-2016
08:55 PM
1 Kudo
Hi, i am new to HDF but am very interested in NiFi. There are a few questions i would like to ask about NiFi. 1; How does NiFi get the data from IoT devices? My understanding is that they send the data to a folder on the machine running NiFi and it uses a getFile processor to retrieve it. Is this incorrect. 2: Can you perform enough processing on NiFi that would ease the workload on other technologies such as spark etc, enabling them to perform tasks quicker? 3: Where does MiniFi fit into the NFi scenario? I have read many documents and watched videos but still need a litlle help in understanding certain aspects of NiFi. I would appreciate any input on these questions, thank you.
... View more
Labels:
- Labels:
-
Apache NiFi
11-21-2016
11:21 AM
Hi @Landon Robinson @jwitt i was trying to install ambari-agent on the pi3 but was having issues as the repo didnt support armv71. is there a way around this, thanks
... View more