About MattWho

MattWho · ‎01-02-2019

You can run the tls-toolkit in client mode directly from any node, but you will either need to provide the CA server info or copy CA config.json to each node manually. I was not trying to imply that you must execute client mode form same server where CA server was installed. - The NiFi CA was not built with the intent for use in a production environment. It was built as a tool that allows users to easily and quickly setup secured NiFi instances/clusters for development and testing purposes. For production environments a corporately/privately managed CA should be used. - There should only ever be one NiFi CA installed and being used to sign all certificates. I apologize if what i wrote was confusing and led you to believe multiple NiFi CAs were needed or should be used. - Feel free to open an Apache NiFi Jira to add the ability to update an existing or output a new nifi.properties file when client mode is used. I don't see that as a bad request at all. - Thank you, Matt

MattWho · ‎01-02-2019

@Nimrod Avni - The Standalone option is not ideal for setting up a NiFi cluster. Since the certificates generated are not signed by a Certificate Authority, the truststore will need to contain a trustedCertEntry for each certificate created. Adding additional nodes to a cluster would require going back and modifying the truststore on every node in the cluster. - The Client/Server mode allows you to standup a Certificate Authority (Server mode) that will be used to sign all the client certificate created (one for each NiFi node). When you stand up the Server a config.json is generated which can be used as input to the client mode operation. Because of this it is common that each of the client certificates are also generated from same server where the CA (server) was created/started. The client mode simply outputs a config.json file for each client certificate which simply provides the information needed to setup the relevant nifi.properties properties on each of your NiFi nodes. - It is safe to say that the structure will remain unchanged with a major NiFi release version. An external script could be used to update a nodes nifi.properties file from the output generated in the client mode config.json file. HDF for example already does this. If you choose to utilize the NiFi CA in HDF, it will take care of obtaining the client certificates and updating the nifi.properties on each node. This allows new client certificate to be generated on demand for each node. There is no option to configure NiFi to read these security parameters from the client mode generated config.json file. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎01-02-2019

@john y - The rest-api endpoint you are using is incorrect for instantiating an existing template on the canvas. You should instead be using a curl command that looks something like this: # curl 'http://localhost:8080/nifi-api/process-groups/<PROCESS GROUP UUID>/template-instance' -H 'Content-Type: application/json' --data-binary '{"templateId":"<THE_TEMPLATE_UUID>","originX":100,"originY":100,"disconnectedNodeAcknowledged":false}' —compressed - The rest-api endpoint contains the UUID of the process group in which you will be instantiating your template. You need to include a header like above that defines the content type and then provide "--data-binary" json that includes the template's UUID and the x coordinates on the graph where the template should be placed. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎12-26-2018

@Ajay Sachdev Both answer provided already are good. Let me explain why they are good here: - NiFi is a flow based programing tool. While NiFi's core itself requires very little resources (CPU and Memory) to run, every user of NiFi builds their own unique dataflow(s) on the NiFi canvas which will have their own unique resource impact/requirement. - Even knowing exactly which processors you will be using, how many of each, and volume/rate of data passing through each would not allow anyone to "exactly" calculate the resource footprint of your dataflow(s). The configuration of these components (processors, connections, controller services, reporting tasks, etc.) and the core (connection swap thresholds, status history retention, etc.) will also impact resource utilization. - It is best to design your dataflow(s) and test the resource impact yourself. NiFi provides some processors like "GenerateFlowFile" which can help you test your flows under load volumes. - Thank you, Matt

MattWho · ‎12-26-2018

@Yahya Najjar Changing the value for "Initial Admin Identity" in your authorizers.xml after NiFi had already been started previously with a different value will not work. - On NiFi startup, the authorizer looks to see if the following files defined in your authorizers.xml already exist: <property name="Authorizations File">/<some path>/authorizations.xml</property><property name="Users File">/<some path>/users.xml</property> If they do exist, they will be used for authorization and new versions of these files will not be generated. - Even though you changed the "Initial Admin Identity" property value, it will result in any change being made to the above files. Since your existing setup already works with your original user certificate, I would suggest authenticating to your NiFi with that certificate and then adding your new user "admin" via the users UI from within the NiFi interface. - You can always delete/rename the existing "users.xml" and "authorizations.xml" files so that new copies are generated on startup, but keep in mind that you will lose and authorizations you may have manually added through the NiFi interface. Another option is to manually edit the "users.xml" file and change the original user based on your user certificate value to the new "admin" value. A restart would be needed for this change to be read in by NiFi. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎12-17-2018

@Eric Lloyd You may want to look in to using the SplitRecord processor instead of SplitContent. You could use a the GrokReader to split you log input. Here is a great article that includes a sample grok pattern for nifi log's format: - https://community.hortonworks.com/articles/131320/using-partitionrecord-grokreaderjsonwriter-to-pars.html - Thank you, Matt

MattWho · ‎12-13-2018

@Allan Batista - Each NiFi node in your 10 node cluster is running with its own copy of the flow.xml and operating against its own set of FlowFiles. Concurrent tasks setting on a component (for example processor) is applied to every node. In order for a NiFi node to execute a component it must have the ability to request a thread from the controller, so it must have at least 1 concurrent task. So setting 1 concurrent task (default) on a processor means that every node can run that processor with one concurrent task (so 10 potential threads in operation across cluster). Threads are not shared between nodes. - Also keep in mind that setting 6 concurrent tasks on a single component (60 equivalent across c10 node cluster) does not mean that that component on each node will always use all 6. If the volume of data being processed by the component does not warrant using more then 1 or 2 concurrent tasks, the other concurrent tasks will not be utilized. - This article may help here: https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor.html - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎12-12-2018

@Gillu Varghese Perhaps the load distribution capability added to connections can help you here: https://blogs.apache.org/nifi/entry/load-balancing-across-the-cluster - You would want to use the either the "Partition by Attribute" or "Single Node" strategy. This would allow you to move all 50 files to a single node for your mergeContent processor. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎12-12-2018

@Puneet Nagpal - You can create an account at https://jira.apache.org/jira/secure/Dashboard.jspa Then you will be able to "Create" "new feature" and "improvement" Jiras against Apache NiFi. The more detail and context you provide in your request to better chance it has of getting added. - Apache NiFi is open source, so feel free to contribute as well. - Thank you, Matt

MattWho · ‎12-11-2018

@Puneet Nagpal - *** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion. - I understand what you are looking for, but unfortunately there is no existing function that can be used to accomplish this.

Online	Offline
Last Visited	‎12-23-2025 01:32 AM

Member Since	‎07-30-2019 10:41 AM
Last Visited	‎12-23-2025 01:32 AM
Posts	3,406
Kudos received	1618

Cloudera Community

Re: Error importing NiFi workflow template from ve...

Re: Error importing NiFi workflow template from ve...

Re: How to elevate a default nifi user to admin - ...

Re: NiFi EnvokeHTTP - putting current date on HTTP...

Re: Invoking Nifi rest api in Data Flow

Re: nifi security configuration

Re: nifi security configuration

Re: Truoble with creating templates and processors...

Re: How does NiFi handle large volume of data (e.g...

Re: NiFi: Unknown user with identity 'admin'. Cont...

Re: SplitContent is adding a newline in Nifi -why?

Re: Limit Concurrent Tasks on Nifi Cluster

Re: How to count the flowfiles from incoming queue...

Re: Evaluate attribute value containing expression...

Re: Evaluate attribute value containing expression...