Cloudera Community
MattWho
user rank
user rank
Master Mentor

Re: How to get only unique data from flow files

by Master Mentor Master Mentor in Support Questions
‎08-05-2016 02:48 PM
‎08-05-2016 02:48 PM
The attached images do not really show us your complete configuration. Can you generate a template of your flow through the NiFi UI and share that? You create a template by highlighting/selecting all components you want to include in your template and then click on the "create template" icon in the upper center of the UI. After the template has been created you can export it out of your NiFi from the template management UI icon (upper right corner of UI). Then attach that exported xml template here. ... View more

Re: How to get only unique data from flow files

by Master Mentor Master Mentor in Support Questions
‎08-05-2016 01:50 PM
‎08-05-2016 01:50 PM
With a NiFi cluster, every node in that cluster runs the exact same dataflow. Some data ingest type processors are not ideally suited for this as they may complete or pull the same data in to each cluster node. In cases like this it is better to set the scheduling strategy on these processor to "On primary Node" so that the processor only runs on one node (primary node). You can then use dataflow design strategies like RPGs (NiFi Site-to-Site) to redistribute the received data across all your NiFi cluster nodes for processing. ... View more

Re: How to get only unique data from flow files

by Master Mentor Master Mentor in Support Questions
‎08-05-2016 11:57 AM
1 Kudo
‎08-05-2016 11:57 AM
1 Kudo
@Yogesh Sharma Is your NiFi a cluster or Standalone instance of NiFi? If it is a cluster, it could explain why you are seeing duplicates since the same GetTwitter processor would be running on every Node. Matt ... View more

Re: Issue in nifi clustering

by Master Mentor Master Mentor in Support Questions
‎08-03-2016 10:55 AM
5 Kudos
‎08-03-2016 10:55 AM
5 Kudos
@Ankit Jain When A NiFi instance is designated as a node its starts sending out heartbeat messages after it is started. Those heartbeat messages contain important connection information for the Node. Part of that messages is the hostname for each connecting node. If left blank Java will try to determine the hostname and in many cases the hostname ends up being "localhost". This may explain why the same configs worked when all instances where on the same machine. Make sure that all of the following properties have been set on everyone of your Nodes: # Site to Site properties nifi.remote.input.socket.host= <-- Set to the FQDN for the Node musty be resolvable by all other instances. nifi.remote.input.socket.port= <-- Set to unused port on Node. # web properties # nifi.web.http.host= <-- set to resolvable FQDN for Node nifi.web.http.port= <-- Set to unused port on Node # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address= <-- set to resolvable FQDN for Node nifi.cluster.node.protocol.port= <-- Set to unused port on Node nifi.cluster.node.protocol.threads=2 # if multicast is not used, nifi.cluster.node.unicast.xxx must have same values as nifi.cluster.manager.xxx # nifi.cluster.node.unicast.manager.address= <-- Set to the resolvable FQDN of your NCM nifi.cluster.node.unicast.manager.protocol.port= <-- must be set to Manager protocol port assigned on your NCM. Your NCM will need to be configured the same way as above for the Site-to-Site properties and Web properties, but instead of the "Cluster Node properties", you will need to fill out the "cluster manager properties": # cluster manager properties (only configure for cluster manager) # nifi.cluster.is.manager=true nifi.cluster.manager.address= <-- set to resolvable FQDN for NCMnifi.cluster.manager.protocol.port= <-- Set to unused port on NCM. The most likely cause of your issue is not having the host/address fields populated or trying to use a port that is already in use on the server. If setting the above does not resolve your issue, try setting DEBUG for the cluster logging in the logback.xml on one of your nodes and the NCM to get more details: <logger name="org.apache.nifi.cluster" level="DEBUG"/> ... View more

Re: Apache Nifi - Adding attributes as newlines Fl...

by Master Mentor Master Mentor in Support Questions
‎08-02-2016 05:48 PM
3 Kudos
‎08-02-2016 05:48 PM
3 Kudos
@Obaid Salikeen Try using \\n (double backslash) or using 'Shift + enter" in the expression language editor box to create new lines in your replacement string as shown by Joe Witt above. Thanks, Matt ... View more

Re: How can I save my NiFi flows into XML?

by Master Mentor Master Mentor in Support Questions
‎07-22-2016 12:26 PM
‎07-22-2016 12:26 PM
@Manikandan Durairaj Simon is completely correct above; however, I want to add a little to his statement about saving the entire flow.xml.gz (Standalone or NiFI Cluster Node) file or flow.tar (NiFi Cluster NCM) file. When you generate templates in NiFi, those dataflows are scrubbed of all encrypted values (passwords). When importing those templates in to another NiFi, the user will need to repopulate all the processor and controller tasks passwords manually. Saving off the flow.xml.gz or flow.tar file will capture the entire flow exactly as it is, encrypted sensitive passwords and all. NiFi will not start if it cannot decrypt these encrypted sensitive properties contained in the flow.xml. When sensitive properties (passwords) are added they are encrypted using these settings from your nifi.properties file: # security properties # nifi.sensitive.props.key= nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL nifi.sensitive.props.provider=BC In order to drop your entire flow.xml.gz or flow.tar onto another clean NiFi, these values must all match exactly. Thanks, Matt ... View more

Re: Why does NiFi GetFile work with KeepSourceFile...

by Master Mentor Master Mentor in Support Questions
‎07-18-2016 10:39 PM
1 Kudo
‎07-18-2016 10:39 PM
1 Kudo
@gkeys What are the permissions on both the file(s) you are trying to pickup with the GetFile processor and the permissions on the directory the file(s) live in? -rwxrwxrwx 1 nifi dataflow 24B Jul 18 18:20 testfile and drwxr-xr-- 3 root dataflow 102B Jul 18 18:20 testdata With the above example permission, I reproduce exactly what you are seeing. If "Keep Source File" is set to true, NiFi creates a new flowfile with the content of the file. If "Keep Source File" is set to false, NiFi GetFile yields because it does not have the necessary permissions to delete the file from the directory. This is because the write bit is required on the source directory for the user who is trying to delete the file(s). In my example nifi is running as user nifi, so he can read the files in the root owned testdata directory because the directory group ownership is dataflow just like my nifi user and the dir has r-x permissions. fi i change that dir permissions to rwx then my nifi user will also be able to delete the testfile. Thanks, Matt ... View more

Re: Connect a Nifi service which lives on an EC2 i...

by Master Mentor Master Mentor in Support Questions
‎07-18-2016 10:09 PM
1 Kudo
‎07-18-2016 10:09 PM
1 Kudo
You could also modify the local /etc/hosts file on your ec2 instances so that the hostname "ip-10-40-197.ec2.internal" resolves to the proper external IP addresses for those zk nodes if they have them. ... View more

Re: The Remote Process Group has warning in an ant...

by Master Mentor Master Mentor in Support Questions
‎07-18-2016 02:52 PM
2 Kudos
‎07-18-2016 02:52 PM
2 Kudos
NiFi secure cluster and Site-To-Site authentication is not handled by kerberos. NiFi kerberos authentication is only supported for user authentication. Secure NiFi Site-To-Site communications are still handled using TLS mutual authentication. The error you are seeing is because that TLS mutual auth is failing. The URL you are providing the Remote Process Group (RPG) is using the IP of the target NCM. The NCM is providing its public key to your nodes for autentication and that certificate does not contain the IP as its DN or as a Subject Alternative Name (SAN). So the source NiFi is saying the that the provided certificate shoudl contain 10.110.20.213 but instead it is providing something else. If you do a verbose listing on your keystore on the NCM you will see the contents of the key. Look for CN=<some value> (This value is typically the hostname/FQDN.) Use that value in the URL you are providing your RPG. Make sure your source NiFi (In your case every Node in your NiFi cluster) can resolve that hostname to its proper IP. The other option is to get a new certificate that has the IP added to it as a SAN. Thanks, Matt ... View more

Re: Productionizing Apache Nifi

by Master Mentor Master Mentor in Support Questions
‎07-13-2016 12:33 PM
1 Kudo
‎07-13-2016 12:33 PM
1 Kudo
i recommend setting up a NiFi cluster that will spread the load across multiple resources. This removes that single point of failure caused by only having one ec2 instance running a NiFI. Now whether a single ec2 instance with NiFi can run your dataflows really depends a lot on your data and what your specific dataflows looks like.for example are you doing a lot of CPU or memory intensive processing in your NiFi dataflows? A good approach is having NiFi sitting on edge systems feeding a central NiFi processing cluster. ... View more
Contact Me
Community Statistics
Member Since ‎07-30-2019 10:41 AM
Last Visited
Posts 3,123
Kudos received 1559