@PradNiFi1236  Another option might be to have two listFile processors. ListFile one is configured with a file filter so that it is only looking for the trg file.  Once the .trg file is listed it feeds an InvokeHTTP processor the you use to start listFile two processor via NiFi rest-api call that is configured to list all the files including the .trg file.  Then ListFile two feeds FetchFile to get each files content.  Then somewhere in this dataflow you use another invokeHTTP processor to invoke a NiFi rest-api call stop listFile two processor. So you have two different dataflows in above example.  With one watching for the triger file and using it to start dataflow 2.   REST API - NIFi 1.26.0 REST API - NiFi 2.x -------- Another option requires you to create a custom processor or use a scripting processor to perform a complete listing when a trigger file is received.  The trigger file comes from am upstream processor like ListFile (configured to only consume .trg files).  The trg file in conjunction with "path" attribute is used in your custom processor to list all files from that target path.    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@carlosst  With a NiFi cluster, each node runs its ow copy of the flow.json.gz into memory on startup. As such processors like listeners will when started create a listener on each node. http://{NiFi node hostname}:{port}/contentListener There is no property to set  the hostname as NiFi uses the hostname unique to each NiFi node here.  All you configure is the port (must be an unused, non-privileged (>1024) ) and Base path (default set to "contentListener"). So any requests received by the listener on a specific node will be processed by that specific node unless you programmatically via your flow redistributed those requests to other nodes via load balanced configured connection. What is commonly done here is to have an external load-balancer in front of your NiFi cluster that handles distributing requests across all your listeners running on the different cluster nodes.  NOTE: when using an external load-balancer in front of NiFi's UI URL, you must configured session affinity (sticky sessions) in that load balancer.  This is not necessary if you are only using the external LB for endpoints like listener since those end points do not use token based authentication) Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Sunny9  What version of CDP is installed?  Is it CDP 7.1.7 or newer as required by CFM 2.1.7? Was parcel and CSDs installed on CM host? Did you verify proper permissions and ownership on the CFM Parcels and CSDs? Parcels are typically owned by root with 755 permissions and are located in /opt/cloudera/parcels/ folder. CSDs are owned typically by "cloudera-scm" user with 644 permissions. (there are two jars for CFM: 1 NiFi and 1 NiFi-Registry).  Add-on CSDs are added typically to /opt/cloudera/csd/ folder.  Make sure you did not add them  to the /opt/cloudera/cm/csd/ folder by mistake. Make sure you have Distributed and Activated the CFM parcel from within Cloudera Manager. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Ali_12012  The documentation for InvokeHTTP states that only POST, PUT and PATCH http methods will sent with a body.  The processor does not support sending a body with GET http method.  Only supports headers. You may need to build a custom processor for your use case or perhaps use one of the scripting processors to accomplish your use case. Thank you, Matt ... View more

@kellerj  CFM has several Service pack versions released for 2.1.5, as well as newer CFM 2.1.6 and CM 2.1.7 versions. If you open the cluster UI via the NiFi UI --> global menu upper right corner) and then click on the "View Details" icon to far left of node that is disconnecting, what Node Events are being reported? Matt ... View more

@enam Have a slight mistake in my NiFi Expression Language (NEL) statement in my above post. Should be as follows instead: Property = filename Value = ${filename:substringBeforeLast('.')}-${UUID()}.${filename:substringAfterLast('.')} Thanks, Matt ... View more

@Vikas-Nifi  the following error is directly related to failure to establish certificate trust in the TLS exchange between NiFi's putSlack processor and your slack server: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target " The putSlack processor utilizes the StandardRestrictedSSLContextService to define keystore and truststore files the putSlack processor will use.  The truststore must contain the complete trustchain for the target slack server's serverAuth certificate. You can use: openssl s_client -connect <companyName.slack.com>:443 -showcerts to get an output of all public certs included with the serverAuth cert. I noticed with my slack endpoint that was not the complete trust chain (root CA certificate for ISRG Root X1 was missing from the chain). You can download the missing rootCA public cert directly from let's encrypt and add it to the truststore set in the StandardRestrictedSSLContextService. https://letsencrypt.org/certificates/ https://letsencrypt.org/certs/isrgrootx1.pem https://letsencrypt.org/certs/isrg-root-x2.pem You might also want to make sure all intermediate CAs are also added and not just the intermediate returned by the openssl command just in case server changes that you get directed to. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@greenflag  Not knowing anything about this rest-api endpoint, all I have are questions. How would you complete this task outside of NiFi? How would you accomplish this using curl from command line? What do the REST-API docs for your endpoint have in terms of how to get files?  Do they expect you to pass the filename in the rest-api request? What is the rest-api endpoint that would return the list of files? My initial thought here (with making numerous assumptions about your endpoint) is that you would need multiple InvokeHTTP processors possibly.   The first InvokeHTTP in the dataflow hits the rest-api endpoint that outputs the list of files in the endpoint directory which would end up in the content of the FlowFile.  Then you split that FlowFile by its content so you have multiple FlowFiles (1 per each listed file).  Then rename each FlowFile using the unique filename and finally pass each to another invokeHTTP processor that actually fetches that specific file. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@NIFI-USER  Are you seeing same behavior even when not using retry strategy of "yield"? What about when retry is not checked?  FlowFiles, upon failure, should immediately be transferred to the connection containing the failure relationship. What are your penalty and yield settings set to on your PublishKafkaRecord_1_0? What version is your target Kafka (you are using a rather old Kafka client version 1.0)? As far as your Kafka topic goes, how many partitions on the topic? How many concurrent tasks set on PublishKafkaRecord? How many nodes in your NiFi cluster? Thanks, Matt ... View more

@Heiko  Thanks for sharing.  The choice between "USE_USERNAME" and "USE_DN" needs to be evaluated against the specific structure of the end user's LDAP/AD. With AD, the user commonly logs in with their sAMAccountName and very often the sAMAccountName value is not the same string used within the user's DN.  While you would still be able to login using your sAMAccountName and password, the user identity passed to the authorizer would be the CN value form that full DN (Your regex assumes the CN consists of only upper or lower case letters and numbers which may not work for all DNs).  Then with the switch to using the CN from the DN, you need to consider equivalent changes in the ldap-user-group-provider in authorizers.xml.  You'll need to make sure whatever user identity strings come out of authentication through DN are properly mapped to group identities. Both solutions will work and both solutions need careful evaluation to setup.  I typically find using USE_USERNAME more consistent in structure (LDAP and AD), and thus less impacted by corner case oddities that using USE_DN can introduce. Thanks again for your contributions to the community.  There is often more then 1 way to solve most queries in Apache NiFi. Matt ... View more