Member since
07-30-2019
3090
Posts
1543
Kudos Received
899
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
94 | 10-31-2024 06:33 AM | |
174 | 10-31-2024 06:07 AM | |
190 | 10-23-2024 09:50 AM | |
201 | 10-23-2024 06:40 AM | |
320 | 10-23-2024 06:33 AM |
10-21-2024
05:41 AM
@Tanya19 @MaxEcueda The PutIceberg and PutIcebergCDC processors only offer Hadoop or Hive Catalog Service provider options currently. The only mention of Glue Catalog i could find in an Apache NiFi JIra was the following still open Jira: https://issues.apache.org/jira/browse/NIFI-11449 It might be a good idea to create an Apache NiFi jira with as much detail as you can provide around this improvement request for an additional AWS Glue Catalog provider. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-18-2024
07:51 AM
@Kiranq This error shared: 2024-10-17 08:35:19,764 ERROR [Timer-Driven Process Thread-5] o.a.n.c.s.StandardControllerServiceNode StandardControllerServiceNode[service=CSVRecordLookupService[id=a8b84b00-b0ee-31c8-dbda-7e7e9795ba4b], name=CSVRecordLookupService, active=true] Encountering difficulty enabling. (Validation State is INVALID: ['CSV File' is invalid because CSV File is required, 'Lookup Key Column' is invalid because Lookup Key Column is required]). Will continue trying to enable. Indicates that NiFi is trying to enable a NiFi Controller services loaded from the flow.json.gz during startup, but cannot because it's configuration is invalid. It is complaining about the configuration of the "CSV File" and "Lookup Key Column" properties. Have you tried starting your NiFi with the following setting in your nifi.properties file set to "false": nifi.flowcontroller.autoResumeState=false This will start NiFi and all components on the canvas will not be started during startup. Also if you NiFi is at the point it is trying to enable components on the canvas, Your NiFi is up and running. As far as the screenshot error, have you verified ownership and permissions on that directory path. Permissions can be an issue if you started the NiFI service as different users at some point in time resulting in some files created on startup having different ownership. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-14-2024
02:54 PM
1 Kudo
@vg27 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
10-10-2024
09:54 AM
2 Kudos
@Krish98 Most NiFi Heap memory issues are directly related to dataflow design. The Apache NiFi documentation for the individual components generally does a good job with reporting "System Resource Considerations". So the first step would be to review the documentation for the components you are using to see which list "MEMORY" as system resource consideration. Example: SplitContent 1.27.0 Then sharing your configuration of those components might help with providing suggestions that may help you. - Split and Merge processor depending on how they are configured can utilize a lot of heap. - Distributed Map cache also resides in HEAP and can contribute to to significant heap usage depending on configuration and the size of what is being written to it. Beyond components: - NiFi loads the entire flow.json.gz (uncompressed it to heap memory). This includes any NiFi Templates (Deprecated in Apache NiFi 1.x and removed in newer Apache NiFi 2.x version). Templates should no longer be used. Any templates created which are listed in the NiFi templates UI should be downloaded so they are stored outside of NiFi and then deleted from NiFi to reduce heap usage. - NiFi FlowFiles - NiFi FlowFlowFiles are what transition between components via connections in your dataflow(s). A FlowFile consists of two parts. FlowFile content stored in content claims in the content_repository and FlowFile metadata/attributes held in heap memory and persisted to flowfile_repository. So if you are creating a lot of FlowFile attributes on your FlowFiles or creating very large FlowFile attributes (like extract content to an attribute), that can result in high heap usage. A connection does have a default threshold at which time a swap file is created to reduce heap usage. Swap files are created with 10,000 FlowFiles in each swap file. The first swap file would not be created until a connection on a specific node reached 20,000 at which point 10,000 would be moved to a swap file and the 10,000 highest priority would remain in heap. The default "back pressure object threshold" on a connection is 10,000 meaning that with defaults no connection would ever create a swap file. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-10-2024
06:04 AM
@Leo3103 What are you seeing in the minifi-app.log? You could also try set DEBUG logging in the minifi logback.xml to capture more details. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-07-2024
07:33 AM
@Axmediko Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
10-07-2024
06:12 AM
@newbeede The only Authentication provider provided with NiFi that supports a single managed user with password is the Single User provider. Apache NiFi create this authentication provider so that NiFi out-of-the-box could run securely over HTTPS. It is intended for non clustered use to make it easy to evaluate the product. The users.xml is created by the file-user-group-provider inside the authorizers.xml. It's is for authorization use only and has nothing to do with user/client authentication. In order to setup various authorization policies through the file-access-policy provider (also found in the authorizers,.xml and generates an authorizations.xml file), the authorizer must be aware of the user/client identities for which policies will be granted. The file-user-group-provider is just one of the available providers that creates these user identities within NiFi for the sole purpose of policy assignment. The file-user-group-provider is used to generate the Initial Admin Identity and the Identities for each of your cluster nodes. The users.xml file it creates will only initially contain those identities. The file-user-group-provider will ONLY generate the users.xml on NiFi startup if it does NOT already exist. Once the NiFi UI is accessed by the initial admin identity, additional user identities and policies are setup from within the UI. Additional user identities and group identities added through the NiFi UI will be added to the users.xml. If you are using the Single-User authentication provider and single-user authorizer, then users and polices will not present in the UI when accessed. The File-access-policy provider sets up admin polices for the defined initial admin identity and sets up initial node required policies. The policies are needed for that initial admin to be authorized to create additional user identities and configure additional policy authorizations from with the NiFi UI. The file-access-policy provider will ONLY generate the authorizations.xml file on NiFi startup if it does NOT already exist. Additional policies setup from with the NiFi UI are added to the authorizations.xml file. So to be clear, NiFi does allow you to dynamically add/remove additional user identities and dynamically add/modify/remove authorizations from the NiFi UI without needing a restart. When it comes to handling user authentication, a secured NiFi will always support client Authentication as first authentication method attempted. This is the only method used for the NiFi nodes to communicate with one another. When additional authentication method are configured, NiFi will still WANT a client auth certificate in the TLS exchange; however if one is not provided, NiFi will try the next authentication method configured. The most commonly used additional authentication method used are the LDAP-provider and Kerberos-provider. The LDAP-provider allows you to authenticate via your own LDAP/AD. Sample: <provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN"></property> <-- MUST be configured
<property name="Manager Password"></property> <-- MUST be configured
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://<hostname>:<port></property> <-- MUST be configured
<property name="User Search Base">CN=Users,DC=example,DC=com)</property> <-- MUST be configured
<property name="User Search Filter">cn={0}</property> <-- MUST be configured
<property name="Identity Strategy">USE_USERNAME</property> <-- MUST be configured
<property name="Authentication Expiration">12 hours</property>
</provider> USE_USERNAME: Upon successful LDAP/AD authentication, the user identity string as typed (case sensitive) in the username login window will be passed to the authorizer to look up policies assigned to the user identity. USE_DN: Will pass the DN returned from LDAP/AD after successful authentication to the authorizer as the user identity for policy assignment lookup. Kerberos-provider allows you to authentication using your kerberos principal and password. Sample: <provider>
<identifier>kerberos-provider</identifier>
<class>org.apache.nifi.kerberos.KerberosProvider</class>
<property name="Default Realm">NIFI.APACHE.ORG</property>
<property name="Authentication Expiration">12 hours</property>
</provider> Bottom line is Apache NiFi does not support managing multiple local accounts. In fact the single-user provider is actually fairly new to Apache NiFi. Prior to its existence, NiFi out-of-the-box started up un-secure over http. Since most browser now force http to https, Apache NiFi added this provider to make it easy for first time user to evaluate the service. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-04-2024
12:20 PM
1 Kudo
@Abibee04 Please start a new Community question with the details of your query. I am not clear what you mean by "recover the registry", so the more detail you can provide the better in that new community question by sharing what step you performed and what you are looking to accomplish. Thank you, Matt
... View more
10-04-2024
09:50 AM
2 Kudos
@varungupta Tell me a bit more about your ListFile processor configuration. Is the input directory a shared mount across all your nodes or does each node have a unique set of files in the input directory? Is ListFile configured for execution of "All nodes" or "Primary node"? How many files are being listed when ListFile is executing (Is it less then 10,000)? How often is ListFile scheduled to run? Is List File traversing sub-directories for files? I assume you are extracting the sequenceNo from the filename. As far as Enforce Order goes: How are you handling the various relationships (routing via connection to where)? Are you seeing FlowFiles routed to "overlook" relationship? Matt
... View more
10-03-2024
02:30 AM
1 Kudo
Set the nifi-registry security INITIAL_ADMIN_IDENTITY value to your admin email address. From the nifi-registry using oidc, login using the admin email address. And you will see all permissions in the nifi-registry page...
... View more