Member since
07-30-2019
3131
Posts
1564
Kudos Received
909
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
106 | 01-09-2025 11:14 AM | |
662 | 01-03-2025 05:59 AM | |
394 | 12-13-2024 10:58 AM | |
428 | 12-05-2024 06:38 AM | |
357 | 11-22-2024 05:50 AM |
08-05-2016
11:57 AM
1 Kudo
@Yogesh Sharma Is your NiFi a cluster or Standalone instance of NiFi? If it is a cluster, it could explain why you are seeing duplicates since the same GetTwitter processor would be running on every Node. Matt
... View more
08-03-2016
10:55 AM
5 Kudos
@Ankit Jain
When A NiFi instance is designated as a node its starts sending out heartbeat messages after it is started. Those heartbeat messages contain important connection information for the Node. Part of that messages is the hostname for each connecting node. If left blank Java will try to determine the hostname and in many cases the hostname ends up being "localhost". This may explain why the same configs worked when all instances where on the same machine.
Make sure that all of the following properties have been set on everyone of your Nodes:
# Site to Site properties
nifi.remote.input.socket.host= <-- Set to the FQDN for the Node musty be resolvable by all other instances.
nifi.remote.input.socket.port= <-- Set to unused port on Node.
# web properties #
nifi.web.http.host= <-- set to resolvable FQDN for Node
nifi.web.http.port= <-- Set to unused port on Node
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address= <-- set to resolvable FQDN for Node
nifi.cluster.node.protocol.port= <-- Set to unused port on Node
nifi.cluster.node.protocol.threads=2
# if multicast is not used, nifi.cluster.node.unicast.xxx must have same values as nifi.cluster.manager.xxx #
nifi.cluster.node.unicast.manager.address= <-- Set to the resolvable FQDN of your NCM
nifi.cluster.node.unicast.manager.protocol.port= <-- must be set to Manager protocol port assigned on your NCM.
Your NCM will need to be configured the same way as above for the Site-to-Site properties and Web properties, but instead of the "Cluster Node properties", you will need to fill out the "cluster manager properties": # cluster manager properties (only configure for cluster manager) #
nifi.cluster.is.manager=true
nifi.cluster.manager.address= <-- set to resolvable FQDN for NCMnifi.cluster.manager.protocol.port= <-- Set to unused port on NCM. The most likely cause of your issue is not having the host/address fields populated or trying to use a port that is already in use on the server.
If setting the above does not resolve your issue, try setting DEBUG for the cluster logging in the logback.xml on one of your nodes and the NCM to get more details: <logger name="org.apache.nifi.cluster" level="DEBUG"/>
... View more
08-02-2016
05:48 PM
3 Kudos
@Obaid Salikeen Try using \\n (double backslash) or using 'Shift + enter" in the expression language editor box to create new lines in your replacement string as shown by Joe Witt above.
Thanks, Matt
... View more
07-22-2016
12:26 PM
@Manikandan Durairaj
Simon is completely correct above; however, I want to add a little to his statement about saving the entire flow.xml.gz (Standalone or NiFI Cluster Node) file or flow.tar (NiFi Cluster NCM) file.
When you generate templates in NiFi, those dataflows are scrubbed of all encrypted values (passwords). When importing those templates in to another NiFi, the user will need to repopulate all the processor and controller tasks passwords manually.
Saving off the flow.xml.gz or flow.tar file will capture the entire flow exactly as it is, encrypted sensitive passwords and all. NiFi will not start if it cannot decrypt these encrypted sensitive properties contained in the flow.xml. When sensitive properties (passwords) are added they are encrypted using these settings from your nifi.properties file:
# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC In order to drop your entire flow.xml.gz or flow.tar onto another clean NiFi, these values must all match exactly.
Thanks, Matt
... View more
07-18-2016
10:39 PM
1 Kudo
@gkeys What are the permissions on both the file(s) you are trying to pickup with the GetFile processor and the permissions on the directory the file(s) live in? -rwxrwxrwx 1 nifi dataflow 24B Jul 18 18:20 testfile and drwxr-xr-- 3 root dataflow 102B Jul 18 18:20 testdata With the above example permission, I reproduce exactly what you are seeing. If "Keep Source File" is set to true, NiFi creates a new flowfile with the content of the file. If "Keep Source File" is set to false, NiFi GetFile yields because it does not have the necessary permissions to delete the file from the directory. This is because the write bit is required on the source directory for the user who is trying to delete the file(s). In my example nifi is running as user nifi, so he can read the files in the root owned testdata directory because the directory group ownership is dataflow just like my nifi user and the dir has r-x permissions. fi i change that dir permissions to rwx then my nifi user will also be able to delete the testfile. Thanks,
Matt
... View more
07-18-2016
10:09 PM
1 Kudo
You could also modify the local /etc/hosts file on your ec2 instances so that the hostname "ip-10-40-197.ec2.internal" resolves to the proper external IP addresses for those zk nodes if they have them.
... View more
07-18-2016
02:52 PM
2 Kudos
NiFi secure cluster and Site-To-Site authentication is not handled by kerberos. NiFi kerberos authentication is only supported for user authentication. Secure NiFi Site-To-Site communications are still handled using TLS mutual authentication.
The error you are seeing is because that TLS mutual auth is failing. The URL you are providing the Remote Process Group (RPG) is using the IP of the target NCM. The NCM is providing its public key to your nodes for autentication and that certificate does not contain the IP as its DN or as a Subject Alternative Name (SAN). So the source NiFi is saying the that the provided certificate shoudl contain 10.110.20.213 but instead it is providing something else.
If you do a verbose listing on your keystore on the NCM you will see the contents of the key. Look for CN=<some value> (This value is typically the hostname/FQDN.) Use that value in the URL you are providing your RPG. Make sure your source NiFi (In your case every Node in your NiFi cluster) can resolve that hostname to its proper IP.
The other option is to get a new certificate that has the IP added to it as a SAN.
Thanks, Matt
... View more
07-13-2016
12:33 PM
1 Kudo
i recommend setting up a NiFi cluster that will spread the load across multiple resources. This removes that single point of failure caused by only having one ec2 instance running a NiFI. Now whether a single ec2 instance with NiFi can run your dataflows really depends a lot on your data and what your specific dataflows looks like.for example are you doing a lot of CPU or memory intensive processing in your NiFi dataflows? A good approach is having NiFi sitting on edge systems feeding a central NiFi processing cluster.
... View more
07-08-2016
12:07 PM
1 Kudo
@mliem
NiFi components (Processors, RPGs, input/output ports, etc...) are designed to run asynchronous. There is no mechanism built in to NiFi for triggering one processor to run as a result of another processor completing its job.
That being said, everything you can do via the UI can be done as well through calls directly to the NiFi API. You may consider playing around with the capability using the invokeHTTP processor to make calls to the NiFi API to start and stop specific processor at specific points in your dataflow. Once a processor is started it will run retrieving a thread from the controller to do so. Stopping That processor will not kill that thread, the processor will simply not be scheduled to run again and will be in a state of "stopping" during that time frame.. You can not start a processor that is still "stopping". So you want to be careful where you invoke your start and stop actions. (For example, following your "matched" criteria you start the mergeContent and after the mergeContent you invoke the stop of the mergeContent.)
For speed and efficiency's sake, I would look for ways to keep your flow asychronous in design. If you do choose to go this route, I would also build some monitoring into your flow using the monitorActivity processor. This processor can be used to monitor that data continues to flow based upon some configured threshold. If that threshold is exceeded it generates a FlowFile that can be routed to a putEmail processor (as and example) to alert someone that the dataflow is down. This is a safety net so to speak in the event one of your api calls fails for some reason (Network hicup for example). Thanks, Matt
... View more
07-07-2016
04:28 PM
It may be helpful to understand your dataflow better if you can paste a screenshot of the second dataflow you want to alter.
... View more