Ask the Community

I can get a Knox token from the Web UI in CDP Private Cloud, but is it possible to get it from the API without having to use the Web UI?

We have Nifi 2.5.0 and a problem whereby some files seem to be "stuck" after leaving a 3 node cluster, and get "stuck" in a site-to-site connection queue after the cluster.We ingest 60,000 small files out of a series of folders that are nested to a depth of 10 folders.Most files seem to move through the cluster OK, but we get a few that seem to enter the cluster fine , but then have a problem leaving it. Its different files each time and they sit there for 30-60 mins and eventually move on. Its not the same files each time.The network and compute resources are fine, the memory that Nifi is using is about 47% of the JVM maximum. Disk space doesn't seem to be an issue either. I had a look through the nifi-app.log file and nothing appears to show as an issue. Cluster heartbeat is average 10 milliseconds,  GC runs in 50 milliseconds every 30 seconds.  Would welcome some suggestions.Thank you in advance.

I'm exporting several tables and I observe that some files are copied to the target path using DistCp (slow) while other files with some other (fast) meanThere is no evidence of the rational behind the choice but the other odd thing is that even if a table is made up of multiple files, hive starts 1 DistCp for each file instead of passing the entire directoryIs there any option to drive the behaviour?

WARNING Port is being used by CM Agent: 7182ERROR Heartbeating to wblvhvbssuat16.telkom.co.za:7182 failed.

We have VM server on which we have NiFi installed within the container.When we try to connect using sftp command from within the docker it connects fine.docker exec -it nifi bashHowever when we try from NiFi PutSFTP processor, it fails with below errorRouting to failure since unable to transfer FlowFile[filename=TGT_TEST28042026.csv] to remote host sftp.AAAAA.com:net. schmizz. sshj. transport. TransportException: Connection reset- Caused by: java.net.SocketException: ConnectionHow and where do I get more details logs ?Also, is sftp command and NiFi PutSFTP works differently ?This is similar to my previous  posthttps://community.cloudera.com/t5/Support-Questions/SFTP-CLI-works-but-PutSFTP-fails/td-p/413903But this time vendor has whitelisted the IP and we are able to connect from within docker on which Nifi is running.ThanksGautam P

Hello I need help to setup oidc based login mechanism for NiFi. I came across this article https://docs.cloudera.com/cfm-operator/2.10.0/configure-nifi-cr/topics/cfm-op-configure-nifi-cr-oidc-auth.htmlbut I want to know how to do it for some provider other than Keycloak. I have an internal provider. Also when I read this article I saw some terms like Discovery url and "The clientID and clientSecret fields are provided to NiFi in a Kubernetes secret. Create that secret with the following command: kubectl create secret generic oidc-client-secret --from-literal=clientID=[***YOUR CLIENT ID***] --from-literal=clientSecret=[***YOUR CLIENT SECRET***]I am quite new to these terms and wanted to know how to do it in NiFi. Is kubernetes required? I have installed both NiFi 1.23 and NiFi 2.8 on a simple machine.

Facing the below issue on Impala - Failed to get minimum memory reservation of 429.94 MB on daemon hydwtbdn1.icicisecurities.corp:22000 for query b44fc02d59517c5b:40ea915f00000000 due to following error: Failed to increase reservation by 429.94 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=255.00 GB reservation=255.00 GB used_reservation=0 child_reservations=255.00 GB The top 5 queries that allocated memory under this tracker are: Query(2e4e69d2f55f0014:fe08e92f00000000): Reservation=153.90 GB ReservationLimit=240.00 GB OtherMemory=1.87 GB Total=155.77 GB Peak=155.85 GB Query(7b478b1891d4039d:8bb6141800000000): Reservation=101.08 GB ReservationLimit=240.00 GB OtherMemory=237.92 MB Total=101.31 GB Peak=101.32 GB Query(e5471eefc1b23d4c:36badde100000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=6.33 MB Total=6.33 MB Peak=925.69 MB Query(6046dfb9cc825b97:45bd0eef00000000): Reservation=0 ReservationLimit

Hello,We are trying to implement Impala Admission Control.Our requirement  is  if we have 5 users let it be A,B,C,D and E. I have created 2  resource pool and added user A,B,C under Submission Access Control of "resource_pool_1" and added user D and E under "resource_pool_2" now we require if user A or B or C will run any impala query it should run in "resource_pool_1" and if user D or E run any impala query it should run in "resource_pool_2". But user does not set pool while running query it should run in allocated pool directlyHow can we achieve this from  Impala Admission Control Configuration?Thanks in advance!

I'm running Nifi in ECS with an ALB. Everything is up and seems normal except there are intermittent 502 errors that pop up in the Nifi ui that look like this It seems that the source of the 502 is the load balancer but since it's intermittent it seems like an interaction between the load balancer and Nifi that is causing this behavior.I've tried messing with the following settings so that the idle timeout (60 seconds) on the ALB is lowernifi.web.request.timeoutnifi.remote.input.http.transaction.ttlI've also made sure the nifi.web.proxy.host configuration is set the hostname that I'm using to hit the loadbalancer. Has anyone else experienced something similar getting a load balancer in front of Nifi?

Previously we used NiFi 1.24, running it in OpenShift containers via the entrypoint "./bin/nifi.sh stateless...". Now I'm trying to migrate the dataflow to version 2.8.0 and am encountering an error - the "stateless" parameter for nifi.sh is now missing. I read all the documentation in the NIFI release notes and did not find information about this anywhere. Git still says that you can run it this way (nifi-stateless-assembly readme).Tell me, is there any possibility to run the NiFi 2.8.0 distribution via the console in stateless mode? How to use a separate nifi-stateless distribution?

Hello I need to invoke a url which is protected by oauth2. I know to invoke simple apis but not with oauth2. How can we make a flow?

Hi everyone,I’m facing an issue while initializing a compute cluster on an existing CDP Public Cloud environment (AWS) using restricted IAM policies.The operation fails with the following error:IAM Restricted Resource Policy validation cannot be completed on AWS:secret encryption is enabled but the secret encryption KMS key is not provided.Liftie does not have the permission to create the KMS key.Please provide a valid Customer Managed Key for secret encryption In the other experience (CDE,CDF) I bypassed this error by adding skip validation. Unfortunately, skip validation is not possible when activating the compute cluster. I've modified also the cross account role by adding the action as mentioned on paragraph "For Let CDP generate CMK" https://docs.cloudera.com/dataflow/cloud/aws-requirements/cdf-aws-requirements.pdf but nothing to do. Do you have any suggestion? Thanks

Hello, everytime I make a new version of or copy 4-5 big process groups, the Nifi UI start throwing 503 errors and sometimes It dies for 15-30 minutes. I tried changing the Garbage Collector algorythm but it did not work out, and I know that adding up memory heap will be useless. What else can I do? Thank you

Hello everyone,during the snapshot for an operational database, we had issues regarding the storage address/location. We were only able to place the snapshot in the same Azure storage container as the HBase service. Otherwise, we got errors for insufficient rights. Are there any official docs helping in this topic?Can Hadoop be modified to allow the use of a different container for storing these snapshots?or how can we give HBase the right to access different storage locations or different storage container?But then, what backups are stored in the default location? Only the cluster backups?Regards(Our Post leading to these questions: https://community.cloudera.com/t5/Support-Questions/HBase-Snapshotting-Failed-because-of-Failure-to-aquire-SAS/td-p/413360)

During testing of the Enterprise Message Bridge implemented using Apache NiFi, it was identified that the Content-Type header becomes null when messages are routed from Message Broker A to Message Broker B.The bridge is designed to enable message flow between the two brokers using Apache NiFi as the intermediary integration layer. However, while validating message headers at the destination broker, the Content-Type attribute is not being preserved and is received as null.This behavior was observed during message inspection at the consumer side connected to Broker B.May I know how to preserve message Content-Type attribute/properties ? Using : ConsumeJMS and PublishJMSProtocol: AMQP 1.0

Have enabled retry option in ListSMB and FetchSMB, since for some reason they are not able to list/fetch files in first attempt.  However, on retry the processor lists and fetches the file successfully.But, the scheduler is skips fews runs.Example:Scheduler is set to run every 30 min and let's assume the job first runs at 00:00 AM.On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully.But, the next scheduler run which is suppose to be at 00:30 AM and 01:00 AM are skipped.Would like to know why retry is impacting scheduler. Thanks.    

SFTP CLI connects to the server and works fine.But, from PutSFTP it fails with below error, this is the only logs we are getting from PutSFTPnet.schmizz.sshj.transport.TransportException: Server closed connection during identification exchangeAlso, both CLI and NiFi are running on same server 

Hello, I am working on a CDP 7.1.9 environment and I am currently testing the Impala ODBC connection using the Cloudera Impala ODBC Driver 2.8.4 on a Unix system with unixODBC. The configuration is as follows: ODBC driver installed and registered in /etc/odbcinst.ini:   [Cloudera ODBC Driver for Impala 64-bit]Description=Cloudera Impala ODBC Driver (64-bit)Driver=/opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.soUsageCount=1   DSN configured in /etc/odbc.ini SSL certificates correctly configured and integrated into the OS trust store Trying to connect using LDAP authentication (username/password)         [Impala_CDP]Driver=/opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.soHost=****************Port=21050Schema=*************AuthMech=3SSL=1TrustedCerts=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pemLogLevel=6LogPath=/tmp/impala2.log However, the connection fails with the following error: isql -v Impala_

How to change the company in the profile? i select wrong company name 

Hello Support Team,We are setting up a CDP Trial environment and noticed that accessing " https://archive.cloudera.com/cdp-public/ "consistently returns an HTTP 404 error, even without authentication. This prevents us from verifying access to public CDP artifacts during the trial installation. Could you please confirm whether the cdp-public path is available for trial users or if it has been deprecated or moved?Best regards,Dae‑Yeol Lee

Hello Everyone,We are currently conducting tests  to read data from CDC Debezium-enabled postgreSQL database and write to Azure EventHub.Actions performed:We uploaded three NARs needed to use the putWe added the following NARs as extensions:nifi-azure-nar-2.4.0.narnifi-azure-services-api-nar-2.3.0.narnifi-cdf-debezium-postgresql-nar-1.28.1.2.1.7.2006-1.narAfter a manifest refresh in EFM, we were able to see and use the PutAzureEventHub and CaptureChangeDebeziumPostgreSQLprocessor processors.We then configured the flow, published a new version, and confirmed that the flow was successfully pushed to MiNiFi.IssueThe Flow is pushed to MiNiFi but it is not being validated, thus the publish request fails, we have errors as if the newly added processors are not recognized, I believe?2026-04-02 12:22:09,379 INFO [MiNiFi logging handler] org.apache.nifi.minifi.StdOut 12:22:09.379 [Validate Components Thread-3] DEBUG org.apache.nifi.controller.AbstractComponentNode

Hello NiFi Community! I have a three-node NiFi cluster. I use this to ingest data from multiple source systems. From time to time, we experience this issue where the flowfiles are getting stucked in round robin queues and just sit idly there.   I've also tried adding funnels just to test if they can still proceed downstream. Upon monitoring the disk usage of the cluster, it never exceeds 30% utilization. Anyone knows what's causing this issue? Tried checking the nifi-app logs but no luck. Thank you!  

HI,I have built a custom processor that uses another java library. This 3rd party JAR requires one sub directory ./data containing a file named config.ini.I put the NAR into the ./extensions subfolder of NiFi. When NiFi starts up, it unpacks the NAR  into the folder nifi/work/nar/extensions/my-processor-nar-1.0.nar-unpacked/This folder contains a subfolder NAR-INF.And this NAR-INF is the location where my subfolder "data" needs to be copied. When I manually copy the folder with the containing config file into NAR-INF, the processor works fine.But after the next restart of NiFi the subfolder is missing again.I need the structure of the unpacked NAR to include this:nifi/work/nar/extensions/my-processor-nar-1.0.nar-unpacked/data/config.iniDoes anyone have an example how I can build the NAR so that this subfolder and file are included?Like an example pom.xml snippet?Cheers,Karsten

We have requirements in a project to Analyze whole workload on Cloudera HDFS data and metadata. We have created a tool which have integration with AWS Redshift, GCP BigQuery and Snowflake. This would be our first on-premise integration in to the tool. Any way I can replicate Cloudera Hadoop on Windows for initial prototype testing without VM which makes system very slow ? RegardsRahul

Hi, I'm having a problem with a NiFi flow. I use UpdateAttrubute processor when I create an attribute like this:  formatDateFrom ${dateFrom:format("yyyy-MM-dd")} After this I have a PutSql processor that changes a status in DB. When everything is fine, it is ok. However, when something is wrong with the change, this status and my flow go to retry this attribute formatDateFrom disappears and is no longer present in the flow. Why does this happen and how can I secure this attribute? Attributes created with EvalueateJsonPath didn't disappear, only those created with UpdateAttribute.