Member since
04-05-2016
139
Posts
143
Kudos Received
16
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
29276 | 02-14-2019 02:53 PM | |
2251 | 01-04-2019 08:39 PM | |
10104 | 11-05-2018 03:38 PM | |
4589 | 09-27-2018 04:21 PM | |
2507 | 07-05-2018 02:56 PM |
01-07-2019
02:43 PM
No problem, glad the advice was helpful. For others following this thread, the Jira filed is: https://issues.apache.org/jira/browse/NIFI-5934
... View more
01-04-2019
08:39 PM
Hi @Yahya Najjar The "operate the component" policy was added to support a role where a user could only start/stop components (https://issues.apache.org/jira/browse/NIFI-375). That can be implemented by removing the user from the "modify the component" policy and adding the user to the "operate the component" policy. What you are trying to accomplish is to have a user that can only modify and not operate, which is not currently supported. The way the "modify the component" policy is implemented, if the user is on that policy, the user can also start/stop components. This supports what is likely the more common scenario: a user who can change a component can also operate it. If you feel NiFi needs to support your use case and would like to see if others in the NiFi community agree, please file an enhancement Jira in the NiFi project (https://issues.apache.org/jira/projects/NIFI/summary).
... View more
11-05-2018
03:38 PM
Hi @Diego A Labrador I believe you need to add the administrator to the "view the data" and "modify the data" access policies for the process group. To be able to see or delete the queue from all connections, select the root process group and click the key icon from the Operate palette: Then select the "view the data" from the Access Policies window drop-down and add your admin to that policy. You should now be able to see the queue in the connection. Similarly, add the admin to the "modify the data" policy to get empty privileges on the queue. More info on these policies can be found: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#component-level-access-policies
... View more
10-26-2018
12:34 PM
Hi @Haider Naveed Have you confirmed that the Controller Services used by the processor (either the Record Reader or Record Reader) are enabled? You can navigate to them using the arrow icon that is next to each on the Properties Tab of ConvertRecord. I wrote a tutorial on using Convert Record (https://community.hortonworks.com/articles/115311/convert-csv-to-json-avro-xml-using-convertrecord-p.html) where you might find some helpful info/screenshots.
... View more
10-24-2018
03:15 PM
2 Kudos
Objective
In a separate HCC article, I detailed how to offload nodes using the NiFi UI. This article covers how to perform the same operation using the NiFi Toolkit CLI.
Note: This tutorial assumes you have already setup a running NiFi cluster. The examples included are for a 2 node NiFi cluster. Environment
This tutorial was tested using the following environment and components:
Mac OS X 10.11.6 Apache NiFi 1.8.0 Apace NiFi Toolkit 1.8.0 Offload Nodes via CLI Tutorial Queue FlowFiles
First, generate some queued flowfiles:
The queued flowfiles are distributed between both nodes in the cluster:
Disconnect Node
From a terminal window, navigate to the directory where the NiFi Toolkit was installed. From the /nifi-toolkit-1.8.0/bin directory, run:
./cli.sh
Enter the command
nifi get-nodes to retrieve the node IDs:
#> nifi get-nodes
# Node ID Node Address API Port Node Status
- ------------------------------------ ------------ -------- -----------
0 dfa62636-479b-4456-aa16-3bfc19f35cb5 localhost 9443 CONNECTED
1 20a68479-e8dc-46c9-8612-872200e2fdab localhost 9444 CONNECTED
Note: If you see the error "ERROR: Error executing command 'get-nodes' : Error retrieving node status: Unable to view the controller. Contact the system administrator." You need to add "CN=localhost, OU=NIFI" to the "access the controller" view policy.
Use the command
nifi disconnect-node to disconnect the node on Port 9444:
#> nifi disconnect-node -nnid 20a68479-e8dc-46c9-8612-872200e2fdab
Node ID: 20a68479-e8dc-46c9-8612-872200e2fdab
Node Address: localhost
API Port: 9444
Node Status:DISCONNECTED~
The node is disconnected:
Note: If you see the error "ERROR: Error executing command 'disconnect-node' : Error disconnecting node: Unable to modify the controller. Contact the system administrator." You need to add "CN=localhost, OU=NIFI" to both the "access the controller" modify policy.
Offload Node
Use the command
nifi offload-node to offload the flowfiles on port 9444:
#> nifi offload-node -nnid 20a68479-e8dc-46c9-8612-872200e2fdab
Node ID: 20a68479-e8dc-46c9-8612-872200e2fdab
Node Address: localhost
API Port: 9444
Node Status:OFFLOADING~
Repeating the command returns the current status of the node:
#> nifi offload-node -nnid 20a68479-e8dc-46c9-8612-872200e2fdab
Node ID: 20a68479-e8dc-46c9-8612-872200e2fdab
Node Address: localhost
API Port: 9444
Node Status:OFFLOADED~
In the UI, it is confirmed that all the flowfiles have been offloaded to the active node:
Delete & Decommission Node
An offloaded node can be either connected back to the cluster (
nifi connect-node ) or deleted. If the goal is to to decommission the node, use the
nifi delete-node command:
#> nifi delete-node -nnid 20a68479-e8dc-46c9-8612-872200e2fdab
OK
Then after the node is deleted, stop/remove the NiFi service on the host.
... View more
Labels:
10-24-2018
02:56 PM
5 Kudos
Objective
With the release of NiFi 1.8.0, flowfiles that remain on a disconnected node can be rebalanced to other active nodes in the cluster via offloading.
Note: This tutorial assumes you have already setup a running NiFi cluster. The examples included are for a 2 node NiFi cluster. Environment
This tutorial was tested using the following environment and components:
Mac OS X 10.11.6 Apache NiFi 1.8.0 Offload Nodes Tutorial Queue FlowFiles
First, generate some queued flowfiles:
In the Global Menu at the top left, select "Cluster" to see the Cluster Management dialog:
The queued flowfiles are distributed between both nodes in the cluster: Disconnect Node
Select Disconnect for one of the nodes:
The node is disconnected: Offload Node
Select Offload ( ) on the disconnected node. This will stop and terminate all processors and rebalance flowfiles to the other connected nodes in the cluster:
Note: Offload would also stop transmitting on all remote process groups if they were in the flow.
When offloading is finished, all of the queued flowfiles are now on the active node: Delete & Decommission Node
An offloaded node can be either connected back to the cluster or deleted. Select Delete:
Once deleted, the node cannot be rejoined to the cluster until it has been restarted. To decommission the node, stop/remove the NiFi service on the host.
... View more
Labels:
10-12-2018
02:30 PM
No problem @Abhilash Dasari, hope you are successful in getting PutDatabaseRecord working. Some PutDatabaseRecord examples in HCC (MySQL database not Teradata): https://community.hortonworks.com/articles/113941/change-data-capture-cdc-with-apache-nifi-version-1-1.html https://community.hortonworks.com/questions/174129/loading-a-simple-csv-file-using-nifiputdatabaserec.html Lastly, the UI in HCC isn't helpful in promoting this, but to keep these question/answer threads cleaner, it is a good practice to select "Add comment" instead of creating a new "Answer" when replying. Thanks!
... View more
10-11-2018
08:27 PM
1 Kudo
Hi @Abhilash Dasari The PutDatabaseRecord processor using a CSV Record Reader looks to be a possible solution: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.1/org.apache.nifi.processors.standard.PutDatabaseRecord/index.html
... View more
10-08-2018
02:13 PM
@Alex Coast A good place to start understanding these properties is in the NiFi Admin Guide: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#back-pressure
... View more
10-01-2018
02:40 PM
Hi @Vijay Mathew It isn't exactly the same issue as yours, but there are some relevant comments in https://issues.apache.org/jira/browse/NIFIREG-197: "there is also metadata for buckets and flows that is stored outside of the GitFlowPersistenceProvider. Currently, to migrate flows it is recommended to use the NiFi CLI, which has export, import, and transfer commands."
... View more