@yoonli  This thread is growing in to multiple queries that are not directly related.  Please start a new community question so the information is easier for our community members to follow when they have similar issues. Thank you, Matt ... View more

@VVPeter  I'd encourage you to create an Apache NiFi Jira with all you details: https://issues.apache.org/jira/browse/NIFI There was an improvement made to the bitbucket registry client in version 2.5 https://issues.apache.org/jira/browse/NIFI-14583 I don't see any direct correlation to your issue., but you could try upgrading to see if your issue persists before raising your bug jira.   Thanks, Matt ... View more

@yoonli  I see three issues: Issue 1: You are using the wrong composite provider.  In my common setup list I properly state you need to be using the "composite-configurable-user-group-provider", but I see you are using the "composite-user-group-provider" class. However, some of the confusion comes from the example I copied from the Apache NiFi Documentation here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-file-and-ldap-based-usersgroups While the NiFi doc example uses the "composite-configurable-user-group-provider" (class name and properties correct), the provider "identifier" says "composite-user-group-provider" still, making this confusing. So it looks like you missed the difference in class name. Since the "file-user-group-provider" is a configurable provider (meaning users/groups can dynamically be added and removed via the NiFI UI), it must be called by a provider that supports a configurable provider. So you'll need to switch from using: <userGroupProvider> <identifier>composite-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider> to using: <userGroupProvider> <identifier>composite-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider> The "identifier" can be any string you want, but the "class" must align with the properties.   Issue 2: Another issue I see is a mismatch in your user identity configured in the file-user-group-provider and the user name shown in the logs coming from your user authentication. file-access-policy-provider: cn=nifi,ou=users,dc=nifi,dc=local nifi-user.log (source of truth) cn=nifi,ou=users,dc=baoviet,dc=local These "user identities" do not match. Also keep in mind that the file-access-policy-provider can only seed policies for "user identities" that are being returned by one of your configured user-group-providers. Where are you expecting this user's DN to be returned from?   Issue 3: Your ldap-user-group-provider is still misconfigured.  the following is not a valid configuration in this provider: <property name="User Search Filter">(cn={0})</property> You can only use "{0}" in the ldap-provider login provider within the login-identity-providers.xml file.  This login provider will substitute the {0} with the username entered in the NiFi login UI. The intent of the ldap-user-group-provider is return many user identities from your ldap. the above filter cn={0} would be treated as a literal and result in no results returned. Also keep in mind that you have configured your ldap-user-group-provider to return the ldap value from the "cn" ldap attribute as the "user identity" which is typically not a full user DN, like we see in the nifi-user.log you shared. <property name="User Identity Attribute">cn</property> Also I see you added this property to your ldap-user-group-provider which is NOT a valid property: property name="Identity Strategy">USE_USERNAME</property> The above property only exist in the ldap-provider found in the login-dentity-providers.xml file.  This is where your probably still have this set to "USE_DN" resulting in the full DN "user identity" you are seeing in the nifi-user.log instead off just "nifi" which i assume you are typing as the username in teh Nifi login window.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue(s) or answering your question(s), please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@AlokKumar  I am not 100% clear on yoru use case.  NiFi can interface with the Soap API using the invokeHTTP processor, but I am not clear on what you men by sending soap request to NiFi to be processed and returned?   Can you provide more detail? Maybe you can use HandleHTTPRequest in NiFi to receive yoru request, pass that to an invokeHTTP processor to interact with the Soap API and get a response which is sent to the the HandleHTTPResponse processor that responds to the original request. For more on the invokeHTTP interfacing with Soap API, here is a possible useful post: https://stackoverflow.com/questions/49457764/nifi-how-to-use-invokehttp-processor-with-soap Apparently someone also created a soap processor you could try: https://github.com/apsaltis/nifi-soap Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@PriyankaMondal  There significant differences between the Apache NiFi 1.x and Apache NiFi 2.x major release.  Deprecated and removed processors, controller services, and reporting task components Some components moved to new nars Deprecated and removed NiFi Templates Deprecated and removed NiFi Variable registry. This means that you can not simply move your flow.json.gz from Apache NiFi 1.23.2 to Apache NiFi 2.x.  First you should update your dataflows so they are no longer using any of the deprecated components in Apache NiFi 1.x.  I recommend first upgrading to Apache NiFi 1.28 so you have the latest deprecation logging.  Apache NiFi 1.28 should produce a deprecation log that will tell you all the deprecated components you are currently using on your dataflow canvas.  Take steps to remove these components or replace them with alternative components that are still available in Apache NiFi 2.x. Examples:  JoltTransformJson processor was included in the  nifi-standard-nar in Apache NiFi 1.x , but has moved to a nifi-jolt-nar on Apache NiFi  2.x.  So the class name has changed.  You'll need to add the Apache NiFi 2.x class version of JoltTransformJson processor to canvas and reconfigure  it for your dataflow and delete the ghosted (dashed line around it because NiFi does not know that class)  JoltTransformJson processor.  While you manually changed the version in yoru flow.json.gz, you did not change the class path as needed resulting in the processor still being ghosted. ConvertAvroToJson processor was deprecated in NiFi 1.x and removed in Apache NiFi 2.x.  Would need to replace it with a convertRecord (available in Apache NIFi 1.x and 2.x) configured to use an Avro Reader and Json Writer. NiFi Variable Registry removed.  So if you are using any NiFi variables in your processor configuration in Apache NiFi 1.x, you'll need to modify your dataflows to use NiFi parameters instead (Parameters exist in Apache NiFi1.x and 2.x) Templates were deprecate in NiFi 1.x and were replaced with Flow Definitions. Templates removed in NiFi 2.x. Would need to remove all templates saved in NiFi before moving to Apache NiFi 2.x. Above is just a short list. refer to the deprecation log produced by Apache NiFi 1.28 to see all deprecated processor you may have been using in yoru dataflows. I do wish there was an easier way to move from Apache NiFi 1.x to 2.x, but depending on your use of deprecated features and changed component classes, there may be little too a lot of effort needed to prepare yoru NiFi 1.x datfalows for migration to NiFi 2.x For Cloudera Flow Management license holders: For Cloudera Flow Management NiFi users, Cloudera has built a Cloudera Flow Management Migration Tool that automated many of the migration steps (swapping processors when alternatives exist, changing processor classes to new classes, converting templates to flow definitions, converting NiFi variables to NiFi parameters, etc.  While there is still no direct upgrade possible from Cloudera Flow Management 2.1.7 (Apache NiFi 1.x based) to Cloudera Flow Management 4.10 (Apache NIFi 2.x based), this migration tool takes a lot of the manual work out of preparing your flow.json.gz for the new major release. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Virt_Apatt  I don't know enough about your use case to make any other suggestions. All I know is that your user(s) supply some custom date that you have NiFi add 10 days to before running a Oracle query to get some result set returned to NiFi. NiFi is typically used to build dataflows that are always in the running state, so users do not need to continuously stop, modify component(s), and start a dataflow/component. What is the significance of this "custom date" that starts your dataflow? Is there any pattern to these custom dates? Can the next custom date be derived from the response from the previous Oracle query? How often does this dataflow get executed? Just some examples (there are many NiFi processor components that can fetch content from external sources): You could start your dataflow with a getSFTP or getFile processor that is checks a specific source SFTP server or local directory for a specific filename. In that file is your custom date.  You then build your dataflow to extract that custom date from the consumed file to then execute your oracle query.  This way your NiFi is always running and just waiting for the next file to show up on the SFTP server or in that local directory it keeps checking. Or maybe setup an http lister (ListenHTTP or HandleHTTPRequest) that listens for an http post that contains the custom date needed for your running dataflow.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@yoonli  Yes, your node certificate DNs do not need to be returned by your ldap. The common cluster setup I shared has both the file-user-group-provider (which you would use to define user identities manually) and ldap-user-group-provider (which syncs users and/or groups from your ldap so you don't need to add them manually). So the file-user-group-provider might look something like this in your authorizers.xml: <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN</property> <property name="Initial User Identity 2"></property> </userGroupProvider> Above will create the user identity "CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN" for your node's certificate manually.  it then becomes available for your file-access-policy provider to use to seed the node policies needed.  If each of your NiFi nodes has a unique DN, you would add as many "Initial User Identity <num>" lines as needed to add them all on initial startup.  ***REMINDER: users.xml and authorizations.xml files are ONLY created if they do not already exist.  Edits to file-user-group-provider or file-access-policy provider will not edit preexisting files. The complete  recommended authorizers.xml setup would look something like this: <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN</property> <property name="Initial User Identity 2"></property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">ANONYMOUS</property> <property name="Manager DN"></property> <property name="Manager Password"></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://localhost:10389</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">ou=users,o=nifi</property> <property name="User Object Class">person</property> <property name="User Search Scope">ONE_LEVEL</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=groups,o=nifi</property> <property name="Group Object Class">groupOfNames</property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <userGroupProvider> <identifier>composite-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">composite-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">nifi</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN/property> <property name="Node Identity 2"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers> *** Reminder: More then one user group provider added to the authorizers.xml can NOT return the same user or group identity.  So do not add any user identities you are syncing from ldap-user-group-provider to the file-user-group-provider or NiFi will throw an exception complaining that two providers returned the same identity. Hope this helps clarify.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue(s) or answering your question(s), please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more