Member since
07-30-2019
2906
Posts
1442
Kudos Received
844
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
54 | 04-17-2024 11:30 AM | |
60 | 04-16-2024 05:36 AM | |
36 | 04-15-2024 05:31 AM | |
120 | 04-03-2024 05:59 AM | |
132 | 04-02-2024 01:22 PM |
12-02-2021
08:48 AM
@Clarfim @tiagot Neither NiFi or NiFi-Registry support creating locally managed users for the purpose of authentication in to services. NiFi provides numerous methods for authenticating a user/client. The default which is always enabled is client/user based authentication via a Mutual TLS handshake. Other methods which can be configured in addition toTLS can be found here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication NOTE: As of Apache NiFi 1.14, a new Single User authentication method was introduced so that NiFi could be secure by default out-of-the-box. If you choose to use this provider, it will be the ONLY user that can access this NiFi and will have full admin access. If full multi-user access is desired, you must use another method of user authentication. The "Users" UI that you see in NiFi is used so that NiFi authorization policies can be assigned to users and group strings. It is not used for creating local users with passwords within NiFi that can be used to authenticate into NiFi. It also will not be present in UI if using the Single User authentication provider mentioned in previous note. When you configured the initial admin user string, it is used to by the authorizers.xml (depending on configuration) to initially construct a users.xml (contains user strings and group strings along with established association between them) and authorizations.xml (contains policies assigned to user and group strings) file. Keep in mind that if the users.xml and authorizations.xml files already exist, they will not be updated if you make changes to your initial admin. They are only generated f these files do not already exist. Above example that results in a users.xml if you are using the file-user-group-provider or legacy FileAuthorizer provider in your authorizers.xml. None of the other providers will produce a users.xml and none of the other providers will allow you to manually add new user strings or group strings. The other providers are all used to pull user and group strings from an external source. The File-Access-Policy-provider is used to construct the authorizations.xml file. The Initial admin string set in this provider allows NiFi to seed the authorizations.xml file when generated with the NiFi policies needed for that user to function as an Admin (reminder: file is only generated if it does not already exist). More info on setting up authorization in NiFi can be found here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
12-02-2021
08:19 AM
@Hafiz The ExtractText processor will evaluate a Java regular expression containing capture group against the inbound FlowFile's content. Then creates FlowFile attributes by processor dynamic property name that is assigned the value from the capture group from that Java regular expression. Above would result in FlowFiles with attributes like: Things to keep in mind. SplitText takes the inbound FlowFile and splits it in too many FlowFiles. If you are producing a lot of splits from a single source FlowFile, it will have an impact of NiFi's heap usage during that process. As each Split FlowFile is created, the FlowFile attributes/metadata fro each produced FlowFile (splits) is held in heap memory. Once all splits are created, all those produced Split FlowFiles are committed to the downstream relationship. One on the relationship, NiFi can then swap as needed out of heap memory. NiFi does this to avoid data duplication. Let's say you have a split that is in progress and NiFi dies. Since nothing has been committed to a downstream relationship yet, when NiFi is brought back online, it will reprocess the original FlowFile. You can reduce heap usage by splitting your source File multiple times if it is large (more than 20,000 - 50,000 splits). For example, split by every 5,000 lines in first SplitText and then by every 1 line in second SplitText. While NiFi does not hold FlowFile content in heap memory (Some processor will load content in to heap to execute on that content), FlowFile attributes/metadata is held in heap memory. So the more attributes/metadata exists on a FlowFile, the more heap that FlowFile is going to use. FlowFiles are held in connection between processor components. NiFi has a connection swap threshold that is applied per connection. The default is to produce swap files that contain 10,000 FlowFiles each (these swap files are for FlowFile attributes/metadata and not content since it is not always held in heap). So swap default set in nifi.properties file is 20,000. This means the first swap file for a connection is generated connection reaches 20,000 queued FlowFiles on one node (if multi-node NiFi cluster, swap is per node and not across all nodes) Just keep above in mind when designing dataflows where you are splitting/merging, creating a lot of FlowFile Attributes, or creating FlowFile attributes with large values. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-30-2021
08:25 AM
1 Kudo
@Bakho @tiagot NiFi 1.14.0 is the first Apache NiFi release that starts up secured by default: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.14.0 NOTE: when using the new Single-User-Provider, NiFi does not support setting new policies or adding additional users and groups via the UI. The "single user" by default is authorized for what is needed to build dataflows. So first make sure if you want to support multiple users, you configure your NiFi to use a different login provider or no login provider and use user/client certificates to authenticate via TLS. If you have already done the above, my guess here is that NiFi was launched for the first time before your initial admin identity was added/set. The NiFi authorizers.xml controls authorization of authenticated users. The default setup utilizes the following providers: <userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity"></property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
<property name="Node Group"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer> The file-user-group-provider creates the users.xml file which holds your local created user and group associations (you would set your initial admin in the "Initial User Identity 1" property. The file-access-policy-provider creates the authorizations.xml file and generates authorizations needed for the configured "Initial Admin Identity" property identity. NiFi will only generate these files if they do NOT already exist. If your NiFi was started before you set the properties I described above for both "Initial User Identity 1" and "Initial Admin Identity", the originally generated users.xml and authorizations.xml files would contain no users or authorizations. I suggest verifying the configuration in your authorizers.xml file, removing the existing users.xml and authorizations.xml files, and then restart your NiFi. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-30-2021
06:02 AM
@Jarinek As part of changing the value of a parameter, all the components that are configured to use that parameter must be stopped and restarted. Some components like controller services have dependent components like processors. So to stop such a controller service, the dependent processor must be stopped first. When NiFi requests that a component stop, it transition in to an intermediate stage fo "stopping" where NiFi asks the thread to exit. The UI would reflect component as stopped or disabled even though it is still in the transition stage. But that does not always guarantee a component will ever complete the transition from "stopping" to stopped. Some client libraries used by components and not written by NiFi developers do support interrupting running threads. In scenarios like this the the "stopping" thread kicked off by NiFi stays active against that component until the active processor thread finally exits (if it ever does). And as i said that specific component thread never completing is out of NiFi's core control. Restarting NiFi is only way to kill any threads that do not gracefully exit. Things you can try here: 1. When you try to enable or delete a controller service, I'd expect the exception to provide the component UUID and host where it claims it is running. You could go to the cluster UI and manually "disconnect" the node where it is still running. Then open a web browser to that specific, now disconnected" node's UI. Use the UUID of the component to locate that component and check its status. Try disabling it and if that does not work, try restarting that node only. If disabling the component worked, go back to UI of another node still in cluster and via the cluster UI request that the disconnected node reconnect. While the exception might point out one host as the issue, it may be more than one host. The UI simply returns the first exception. 2. If above is not successful, have you tried restarting the entire cluster? 3. You could also inspect the the flow.xml.gz from each host to make sure they match. Component state (started, stopped, enabled, and disabled) is retained in the flow.xml.gz, but is not included within the flow fingerprint used when connecting a node to the cluster. As a connecting node is suppose to inherit the state from the cluster. You could check each flow.xml.gz for the component UUID and see what its current recorded state is in the flow.xml.gz. If you find one or more with a state of enabled/running and other nodes with an alternate state, simply copy the flow.xml.gz with the desired state to the nodes with the undesired state. Make sure flow.xml.gz ownership and permissions are correct and restart your NiFi cluster. As far as to why a component never transitioned from state to another, that would require examining numerous thread dumps spaced apart by at least 5 minutes. This would allow you to identify active thread making no progress over the course of those numerous thread dumps. Then you would have to to dig in to those threads to see what they are waiting on or blocked by. Sometimes there are limitation within some clients that NiFi has no control over. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-23-2021
08:05 AM
@toninghaywi The exception is implying that your NiFi is trying to connect to Zookeeper server for state management and your zookeeper is using TLS; however, your NiFi has not been configured to be able to establish that secure connection needed to your zookeeper. Apache NiFi support for TLS enabled zookeeper server was added in Apache NiFi version 1.13. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zk_tls_client https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#embedded-zookeeper-with-tls The Apache NiFi jira related to this feature improvement are below: https://issues.apache.org/jira/browse/NIFI-7203 https://issues.apache.org/jira/browse/NIFI-7115 https://issues.apache.org/jira/browse/NIFI-7819 <-- This one is specific to state management If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-23-2021
05:26 AM
1 Kudo
@Protector Do all your json FlowFiles have a FlowFile attribute on them for "table_name". It is not pulling table_name from the FlowFIle content (your json content) itself. The Correlation Attribute Name property in the MergeContent processors is looking for this FlowFile Attribute on each incoming FlowFile in order to allocate those FlowFiles with same value assign to that FlowFile attribute to the same bin. Then a bin is merged when it meets the other configured mins on the MergeContent, max bin age is reached, or all bins have files allocated to them and another bin is needed forcing the merge of the oldest bin. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-23-2021
05:14 AM
@emmanuel Yes, Apache NiFi support PKCS12 keystores; howevere, JKS is more commonly used format and was making a suggestion to test using JKS to rule out an issue there. Something else you may want to do is enable debug logging for TLS in NiFi-Registry. You can accomplish this by adding an additional line to the bootstrap.conf file: java.arg.<unique num>=-Djavax.net.debug=ssl,handshake Maybe you are having a cipher compatibility issue? Maybe connection is trying to use TLS 1.3 and that needs to be disabled so 1.2 is negotiated? Java's default cacerts file is not used by the NiFi core during Mutual TLS negotiation. Only the configured truststore configured in the nifi.properties and nifi-registry.properties file is used. Hope this helps in your investigation, Matt
... View more
11-22-2021
11:01 AM
@HiSunny The configured "run schedule" on a NiFi processor controls how often the processor is scheduled to execute. The "concurrent tasks" controls the parallel execution of a processor. When a processor is scheduled to execute it will request a thread from the NiFi Max Timer Driven thread pool that will be used to execute the processor code. If that thread is still active upon next scheduled Execution and not all concurrent tasks are in use yet, the processor can request another thread to execute in parallel. When it comes to variables, If the processor property support NiFi Expression Language (NEL) with an evaluation using FlowFile attributes, then each FlowFile via FlowFile attributes can provide unique input to the execution. You can see if a property supports Expression language by floating your cursor of the "?" icon next to each property name. You'll want to make sure the scope of NEL support supports FlowFile attributes. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-22-2021
09:19 AM
@emmanuel Does your truststore on each NiFi host contains a separate TrustedCertEntry for each of your three CA certificates in your trust chain? Your openssl command output does not list your root or intermediate CAs: Acceptable client certificate CA names
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost Have you tried converting your PKCS12 keystore to a JKS keystore? Matt
... View more
11-22-2021
09:04 AM
@Venkikancharla You are absolutely correct that CFM 2.1.2 is also impacted by this. While CFM 2.1.2 is based off Apache NiFi 1.13.2, it includes many bug fixes that eventually went in to Apache NiFi 1.14. One of those changes happened to be NIFI-8723. Full list of changes made on top of Apache NiFi 1.13.2 in CFM 2.1.2 can be found here: https://docs.cloudera.com/cfm/2.1.2/release-notes/topics/cfm-fixed-issues.html CFM 2.1.3 will include https://issues.apache.org/jira/browse/NIFI-8938 Thank you, Matt
... View more