Hello @hus,  Directly the ExecuteSQL processor does not support binding.  The input should be directly SQL query.  https://nifi.apache.org/components/org.apache.nifi.processors.standard.ExecuteSQL/ But, according to the API docs, the processor can be started via a FlowFile:  https://javadoc.io/static/org.apache.nifi/nifi-standard-processors/2.4.0/org/apache/nifi/processors/standard/ExecuteSQL.html  So technically you should be able to start this query by using ReplaceText to inject values dynamically, for example.  ... View more

@zzzz77  Provenance can be very noisy depending on size of your dataflows and the amount of FlowFIles being processed through those dataflows.  The provenance repo has age and size configuration that trigger roll-off of old events.   So you may not reach the retention age if you reach size first.  Also would not be trying to read provenance files while they are being written to.    The SiteToSiteProvenanceReportingTask might be the solution you are looking for in Apache NiFi.   This reporting task will send all provenance events over Site-To-Site protocol to a target NiFi where you can then feed them into any long term storage medium of your choice in a human readable format.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Hello @SalimAlhajri,  The Transparent Authentication is built-in Cloudera AI.  When the application is started, it will inject the REMOTE-USER and REMOTE-USER-PERM HTTP headers automatically, this is why it is transparent, no manual intervention is needed.  https://docs.cloudera.com/machine-learning/1.5.5/applications/topics/ml-securing-applications.html  ... View more

Hello @NadirHamburg  Thanks for being part of our Community.  I'm not an expert on Clickhouse, but was reading that it could be something on the DB causing the batches to repeat and causing that amount of duplicated records.  From NiFi side, you can try to set the batch size at the same amount of records, this should work for you. But I know that for big databases it could be a problem.  From Clickhouse, I found this documentation:  https://clickhouse.com/docs/engines/table-engines/mergetree-family  There talks about ReplicatedMergeTree, which should be a good option to avoid duplicates.  Do you have your table with those settings? Do you see any errors on PutDatabaseRecord log? If so, can you share them?  ... View more

Hello @garb,  Thanks for being part of our community.  I was reviewing the information and even though Calcite is the engine used for the SQL, looks like not all the queries are supported officially.  Looking in several places, I do not see LPAD used anywhere.  But something that may work for what you need is CONCAT, which should give you the correct format properly and is broadly used in the community:  SELECT MsgSeqNbr, PostTime, SSN, EmployeeID, LName, FName, MName, CAST(TRIM(LCN) AS BIGINT) AS LCN, RIGHT(CONCAT('00000', PIN), 5) AS PIN, EmployeeType, ValidityCode, AgencyOwner, AgencyLocated, BadgeCreatedBy, BadgeCreatedTime, BadgeModifiedBy, BadgeModifiedTime, Clearance, Error, Status FROM FLOWFILE Based on the error "No match found for function signature LPAD" it looks like the engine configured for NiFi does not support LPAD even when Calcite do support it.  I was trying to find on the code the supported functions, but did not find LPAD.  This looks to be the most accurate reference we have where we do see CONCAT: https://github.com/apache/nifi/blob/main/nifi-docs/src/main/asciidoc/record-path-guide.adoc  ... View more

@zzzz77  FlowFile Metadata/attributes are held in NiFi Heap memory.  For queued FlowFiles, there is a configurable swap threshold in the nifi.properties that will swap batches of 10,000 FlowFIle's worth for metadata/attributes to disk when the threshold is met.  This swapping is there to minimize excessive heap usage when queues grow large.  The NiFi Content is not held in heap memory; however, some processor may need to read the content into heap memory for the processor to perform it's function.    You will notice if you look at the individual components documentation that a "System Resource Considerations" section exists.  If Heap memory usage is a concern for that processor, it will be documented there.  SplitContent processor docs example: Processors like SplitContent will hold the all the FlowFile metadata/attributes (not content) for every split FlowFIle being produced in heap memory until all the output FlowFiles have been produced and committed to the downstream connection.  These FlowFiles being produced can not be swapped to disk until they committed to the downstream connection.  So if a splitContent were to produce 50,000 split FlowFiles, the attributes for all 50,000 would be held in heap.  After committed to the downstream connection. 40,000 of those would get swapped to disk based on default swap thresholds. So heap impact would spike but not persist.  Since you have not shared the specific of your dataflow in question (which processors you are using), I can't provide any specific feedback.   Where is the chunking and de-chunking happening?  Sounds like this may be happening at source and at destination. NiFi is just moving these chunks from source to destination.  How are you sending the chunks to NiFi and transferring them to destination?   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Hello @raghavhinduja26,  The Cloudera documentation have the steps for Ubuntu installation.  Take a look here: https://docs.cloudera.com/cloudera-manager/7.13.1/cloudera-manager-installation/topics/cdpdc-installing-cm-runtime.html On each of the steps you have 3 tabs: RHEL, SLES and Ubuntu.  Follow those steps and you should be good to go.  ... View more

@MattWho  1. I am using java version 21 2. Using Windows 10  Version number (22H2) 3.  I am using Chrome. Tried to use Mozailla it's still same problem. when I click "view content" it open a new tab with request "https://localhost:8443/nifi/#/content-viewer/?ref....." then immediately it changes to "https://localhost:8443/nifi/#/error" I have opened the DevTools and tried to see the response but nothing appears in the response tab  . I am trying to solve this from last week   ... View more

Hi @haridjh I tried to delete the VC using `cdp de delete-vc --vc-id...  --cluster-id...` and got an error result of auth failed: (Status Code: 403; Error Code: PERMISSION_DENIED; Service: de; Operation: deleteVc; Request ID: 94ac6721-5242-43d2-963b-c5b907ca030c;)   ... View more