@hegdemahendra  As far as your issue goes, it would probably be useful to collect a series of thread dumps (at least spaced 5 minutes apart).  Then you would be looking for any threads related to the stopping of components to see if they are progressing or hung.  Is it stuck on stopping a specific processor or processor class?  Do any of the processors that are being stopped have active threads showing for them? Thank you, Matt ... View more

@BK84  I suggest starting a new community question for your new query.   When you start you r new question, please provide more detail on your ask.  Not clear what you mean by "trigger".   What is the use case you are trying to implement? Thank you, Matt ... View more

@Krish98  Not enough details here to determine what is going on.  Do all your FlowFile that are being merged contain the required FlowFile attributes needed for this to be successful? Name Description fragment.identifier Applicable only if the <Merge Strategy> property is set to Defragment. All FlowFiles with the same value for this attribute will be bundled together. fragment.index Applicable only if the <Merge Strategy> property is set to Defragment. This attribute indicates the order in which the fragments should be assembled. This attribute must be present on all FlowFiles when using the Defragment Merge Strategy and must be a unique (i.e., unique across all FlowFiles that have the same value for the "fragment.identifier" attribute) integer between 0 and the value of the fragment.count attribute. If two or more FlowFiles have the same value for the "fragment.identifier" attribute and the same value for the "fragment.index" attribute, the first FlowFile processed will be accepted and subsequent FlowFiles will not be accepted into the Bin. fragment.count Applicable only if the <Merge Strategy> property is set to Defragment. This attribute indicates how many FlowFiles should be expected in the given bundle. At least one FlowFile must have this attribute in the bundle. If multiple FlowFiles contain the "fragment.count" attribute in a given bundle, all must have the same value. segment.original.filename Applicable only if the <Merge Strategy> property is set to Defragment. This attribute must be present on all FlowFiles with the same value for the fragment.identifier attribute. All FlowFiles in the same bundle must have the same value for this attribute. The value of this attribute will be used for the filename of the completed merged FlowFile. Are the FlowFile attribute values correct? How are the individual FlowFile fragments being produced? I find it odd that you say 1 FlowFile is routed to "merged".  This implies the a merge was success.  Are you sure the "merged" FlowFile only contains the content of one FlowFile from the set of fragments? Any chance you routed both the "original" and "failure" relationships to the same connection?  Can you share the full MergeContent processor configuration (all tabs)? When a FlowFile is routed to "failure", I would expect to see logging in the nifi-app.log related to reason for the failure.  What do you see in the nifi-app.log? How many unique bundles are you trying to merge concurrently?  I see you have set "Max Num Bins" to 200.  So you expect to have 200 unique fragment.identifier bundles to merge at one time?    How many FlowFiles typically make up one bundle?  I also see you have "Max Bin Age" set to 300 sec (5 mins).  Are all FlowFiles with same fragment.identifier making it to the MergeContent processor within the 5 minute of one another? Keep in mind that the MergeContent has the potential to consume a lot of NiFi heap memory depending on how it is configured.  FlowFile Attributes/metadata is held in heap memory.  FlowFiles that are allocated to MergeContent bins have there FlowFiles held in heap memory.  So depending on how many FlowFiles make up a typical bundle, the number of bundles being concurrently handled (200 max), and the amount + size of the individual FlowFile attributes, binned FlowFils can consume a lot of heap memory.  Do you encounter any out of memory errors in yoru nifi-app.log (might not be thrown by mergeContent).   You must be mindful of this when designing a dataflow that uses MergeContent processors. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Thanks for the references @MattWho. Currently I've implemented a workaround with counters, and querying the nifi-api with http requests to get around this.  It's definitely not a bullet proof implementation, with corner cases that need to be handled separately, but it's a start to build off of. ... View more

@Shelton  Thank you for your advice. As I use the latest version of NiFi and it requires Java 21, I added the following line in bootstrap.conf and confirmed the warning messages disappeared. java.arg.EnableNativeAccess=--enable-native-access=ALL-UNNAMED  I appreciate your help. Thank you, ... View more

@Zifo1  When using Site-to-SIte via Remote Process Groups (RPG) and Remote Input or Output ports between NiFi clusters, it is most efficient to push rather then pull data (FlowFiles).  The NiFi RPG always acts as the client side of the connection.  It will either send FlowFiles to a Remote Input Port or fetch FlowFiles from a Remote Output port. I would avoid fetching from Remote Output ports.  You get better FlowFiles distribution across teh destination cluster when you send FlowFiles from the RPG. If the FlowFiles traverse both directions, you would simply setup a RPG on both NiFi clusters to push FlowFiles to the Remote Input Ports on opposite clusters. Details about Site-To-Site can be found here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site As far as the RPG goes, I recommend using the "RAW" transport protocol over HTTP.  RAW requires that the dedicated RAW port is configured in the server side NiFi's nifi.properties file.  RAW establishes a raw socket connection on the dedicated configured port.  HTTP utilizes the same HTTPS port that all other NiFi interactions use.  You'll need to make sure the network connectivity exists between both your NiFi Clusters on both the HTTP(s) and RAW ports.  HTTP is always used to fetch Site-to-Site Details.  Setting up the client side (Remote Process Group) Documentation is here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-client-nifi-instance Setting up the sever side (NiFi with Remote Input or Remote Output ports) documentation can be found here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-server-nifi-instance Even with Site-To-Site the communications between the two NiFi clusters requires both authentication and authorization.  Authentication is established via a mutual TLS handshake initiated by the RPG.  For Site-to-Site, the keystore and truststore setup en each NiFi's nifi.properties file are used in the MutualTLS exchange. NOTE: The NiFi out-of-box auto generated keystores and truststores are not suitable for negotiating a successful Mutual TLS handshake. There are numerous authorization policies that must be setup on the server side (remote ports NiFi) so that the client side (NiFi with RPG) is able to successfully send FlowFiles over Site-to-Site: 1. Retrieve Site-to-Site Details - This policy authorizes the client NiFi nodes (so all nodes in the client side NiFi cluster must be authorized) to retrieve site-to-site details from the server side NiFi.  This includes details like number of nodes, load on those nodes, authorized remote ports, site-to-site raw port, https port, etc. 2. Receive data via Site-To-Site - This policy is setup on Remote Input ports to authorize the client side NiFi nodes to send FlowFiles to this specific port. 3. Send data via Site-to-Site - This policy is setup on the Remote Output Ports and allows authorized client nodes to fetch FlowFiles from the Remote output port. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@sha257  Did you restart your NiFi after modifying the logback.xml? NiFi executes the ldap-user-group-provider during startup, so you should see DEBUG output in the nifi-app.log at that time and each time the sync interval happens. As far as ldap sample, I would need to see a ldap search group output that contains "member" entries for users you expect to see. Thanks, Matt ... View more

@SAMSAL  Hi, sorry that I was not clear enough in my question, was doing my best 😞  @MattWho  This works, thank you very much. I have set a maximum back-off time for 1 minute as this process is quite time-sensitive and set the queue before RouteonAttribute as well as looped relationship to prioritize the oldest fileflow. Thank you both for help! ... View more

It works fine @archie ! Thanks! This is my following snipped docker-compose code: nifi-registry: image: apache/nifi-registry:1.23.2 container_name: nifi-registry hostname: nifi-registry restart: "no" ports: - 18443:18443 volumes: - ./nifi/certs/localhost:/opt/certs environment: TZ: America/Sao_Paulo NIFI_REGISTRY_WEB_HTTPS_PORT: 18443 AUTH: oidc KEYSTORE_PATH: /opt/certs/keystore.jks KEYSTORE_TYPE: JKS KEYSTORE_PASSWORD: changeit TRUSTSTORE_PATH: /opt/certs/truststore.jks TRUSTSTORE_PASSWORD: changeit TRUSTSTORE_TYPE: JKS INITIAL_ADMIN_IDENTITY: test@test.com NIFI_REGISTRY_SECURITY_USER_OIDC_DISCOVERY_URL: http://<LOCAL_KEYCLOAK_IP>:8080/realms/TEST/.well-known/openid-configuration NIFI_REGISTRY_SECURITY_USER_OIDC_CONNECT_TIMEOUT: 10000 NIFI_REGISTRY_SECURITY_USER_OIDC_READ_TIMEOUT: 10000 NIFI_REGISTRY_SECURITY_USER_OIDC_CLIENT_ID: nifi NIFI_REGISTRY_SECURITY_USER_OIDC_CLIENT_SECRET: <CLIENT_SECRET> NIFI_REGISTRY_SECURITY_USER_OIDC_PREFERRED_JWSALGORITHM: RS256 NIFI_REGISTRY_SECURITY_USER_OIDC_ADDITIONAL_SCOPES: openid,email,profile NIFI_REGISTRY_SECURITY_USER_OIDC_CLAIM_IDENTIFYING_USER: preferred_username networks: - test-net     ... View more

@pankajgaikwad As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks. ... View more