Ask the Community

Hi everyone, I need some quick help. I was doing some research on a tool that can help streamline the deployment of Apache NiFi flows across multiple environments. Does anyone know of any such tools, whether free or paid?

 Hi Cloudera Community/@@MattWho,I'm currently working with Apache NIFI version 1.28.1 and have successfully configured SAML authentication using an  Okta for UI login.I now need to access the NIFI REST API (/nifi-api) programmatically, but I'm facing challenges due to the nature of SAML being browser-based. Specifically, I’d like to know:What are the supported or recommended methods to access the NIFI API when SAML authentication is enabled?Is there a way to retrieve or reuse the JWT token issued after SAML login for API calls?Can NIFI processors or external scripts be used to automate token retrieval or API access in this setup?Are there any best practices or alternative authentication mechanisms (e.g., client certificates or hybrid setups) that can be used alongside SAML for API access?Any guidance or examples would be greatly appreciated!Thanks in advance.

hi community cloudera,recently, we've been receiving the following error when querying impala on hue:queryerror: tsocket read 0 bytes (code thrifttransport): ttransportexception('tsocket read 0 bytes',)- the "query_timeout_s" and "session_timeout_s" parameters have already been set to 2 hours in the hue settings.- the hs2 and impala thrift connection timeout has already been set to 2 hours in the hue settings.- the hs2 authentication timeout has already been set to kerberos in the hue settings.- the "proxytimeout" parameter has already been set to 10 minutes in the hue settings.and we're still receiving this error.has anyone experienced this issue and managed to resolve it?we're using cdh 6.3.4.

Hello, guys!I still have a three-node cluster Apache NiFi 1.18.0 (yeap, upgrade task in by backlog =)) and trying to setup it behind HAProxy just for UI (/nifi endpoint).I wanna to implement a health check to balance user requests only to nodes in CONNECTED status to avoid issues when nodes moves to maintenance (offload, disconnect).Is it good idea, right? Or maybe I do some overengineering things? 😃By the way, I have next HAProxy config file:frontend nifi_443 bind *:443 ssl crt /etc/haproxy/ssl/cert_file.pem http-request set-path /nifi if { path / } mode http option httplog acl DST_IP dst 10.0.0.10 use_backend nifi_ui_9443 if DST_IPbackend nifi_ui_9443 mode http option httpchk http-check send meth GET uri /nifi-api/access/config ver HTTP/1.1 hdr Host nifi-cluster.corp.company.com http-check expect status 200 cookie SERVERID insert indirect nocache balance leastconn server srv_1 10.0.0.1:9443 check ssl verify none cookie web1 server srv_2 10.0.0.2:9443 check ssl verify

Hello Cloudera Support,We are receiving warnings in Cloudera Manager related to swapping activity on one or more cluster nodes. However, swap has already been disabled across all our hosts, and we've verified that no active swap space is configured. Despite this, Cloudera Manager continues to raise the warning.We would appreciate your assistance in clarifying the following:What conditions or metrics trigger this warning?Are there additional configurations or steps we should take to prevent this alert?Please let us know if any diagnostics or host information is required from our end.Thank you for your support. (Swap has been disabled across all nodes via OS configuration.) Best regards,Modhi Alrashed

Hi Team,             we have an issue with Nifi UI, which is unstable with multiple nodes (like 401 unauthorized error) but with single node it is working fine without any issue, For multiple nodes also UI coming but dropping off after sometime, and accesible to only to few  users randomlycoming to the background of this issue, we recently migrated our application from DHC CASS  to DYP CAAS (AWS), all configurations are fine and backend also working fine. only thing changed is the domain name (hostname changed) we replaced this latest hostname in all config files (users,nifi.properties, autherizations.xml, sts, deployment) wherever it is needed in place of Old domain name.from Browser network logs what we observed is 401 unauthorized error whenever its throwing error logs unauthorized Please check logs. and these errors everytime changing to different api's in the nifi not consistent, sometimes its coming with config, sometimes summary,

ozone FS datanodes disk space is getting occupied immediately even after extend the disk space without any operations performed.

Hi All,Let's say I have a Process Group that runs once per day and contains a set of processors. What I would like to track is:When the Process Group startedHow long it ranWhen it completed...both at the Process Group level and the individual processor level within the group.Can we capture this information from NiFi logs? If these details are not available in the logs, where else can I find them?Basically, I'm working on building a centralized table to store daily run details for each Process Group.

Hello Cloudera Support,We are running an environment based on Apache Ambari, where Apache Log4j 1.x is in use.The identified security vulnerabilities are:CVE-2019-17571 – Insecure deserialization via SocketServerCVE-2020-9488 – TLS certificate hostname validation issue in SMTPAppenderCVE-2022-23302 – Unprotected JNDI usage in JMSSinkSince Log4j 1.x has reached End of Life (EOL), no security updates are available.Additionally, in our Ambari environment, upgrading to Log4j 2 is not supported by Cloudera, and therefore version upgrade is not an option.QuestionsIs there a supported workaround for Log4j 1.x similar to the official solution (cloudera-scripts-for-log4j) you provided for Log4j 2.x vulnerabilities?Would manually removing the vulnerable classes (e.g., SocketServer.class, JMSSink.class, JMSAppender.class, SMTPAppender.class) from the Log4j 1.x JAR files cause any service interruptions or functional issues in Ambari or Hadoop components (HDFS, YARN, HBase, Hive, Kafka, etc.)?Are t

Hi everyone, I recently completed Oracle Exadata training from some excellent instructors and gained a solid understanding of its architecture, smart scan capabilities, and performance optimization features. I’m now looking to expand my knowledge by learning from real-world use cases. Specifically, I’m interested in: Common challenges faced when running Exadata in production Performance improvements compared to traditional DB setups Integration scenarios (especially with Cloudera or big data platforms, if any) If anyone here has worked with Oracle Exadata alongside Cloudera or in a hybrid environment, I’d really appreciate your insights. Also open to any tips on hands-on labs or certification paths worth exploring. Thanks in advance!

Datalake creation failed. Data Lake creation failed on 'cdp-dl-aws-01' cluster. Reason: Stack status: CREATE_FAILED, cluster status: CREATE_FAILED, reason: Installation of CDP with Cloudera Manager has failed. Please find more details on Cloudera Manager UI. Failed command(s): HiveCreateSysDb(id=1546337523): Failed to Create Hive Sys Database.

Hello Everyone,We are currently conducting tests and POCs to push (and potentially read) data from Kafka brokers and Azure Event Hubs.Event Hubs TestOur Event Hubs test worked successfully:We added the following NARs as extensions:nifi-azure-nar-2.4.0.narnifi-azure-services-api-nar-2.3.0.narAfter a manifest refresh in EFM, we were able to use the PutAzureEventHub processor.We configured the flow, published the new version, and confirmed that the MiNiFi agent could successfully send data to the Event Hub.Kafka Test IssueWe are now trying to create a flow in MiNiFi to push data into a Kafka topic.We followed the same steps as the Event Hubs test and added the following NAR files:nifi-kafka-service-api-nar-2.4.0.narnifi-kafka-nar-2.4.0.narWhat works:After refreshing the manifest, we can see two Kafka processors, including PublishKafka, which we need for the test.What doesn’t work:We cannot add the KafkaConnectionService, which is required to configure the PublishKafka processor.Even after

Hi Pros,Current version:Cloudera Manager 7.11.3.0Cloudera Runtime 7.1.7 SP1Upgrade version:Cloudera Manager 7.11.3.14 (CH7)Cloudera Runtime 7.1.9 SP1I'm in the midst of upgrading Cloudera 7.1.7 SP3 to Cloudera 7.1.9 SP1. I encountered error at  Install YARN MapReduce Framework JARs on service YARN. I have screenshot and full logs but unable to upload here due to no permission. Below is an extract of the error logs./user/yarn/mapreduce/mr-framework - have executable rights.Appreciate if anyone could point me to the answers.Thank you very much.25/07/18 12:59:43 ERROR uploader.FrameworkUploader: Error in execution /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/lib/hadoop/./ozone-filesystem-hadoop3-1.4.0.7.1.9.1000-103.jar (Permission denied)java.io.FileNotFoundException: /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/lib/hadoop/./ozone-filesystem-hadoop3-1.4.0.7.1.9.1000-103.jar (Permission denied)at java.io.FileInputStream.open0(Native Method)at java.io.FileInp

Hi Cloudera Community,We are running Apache NiFi version 1.28.1 in a clustered setup with the following specifications:Cluster Size: 6 nodesEach Node: 32 vCPUs, 256 GB RAMJVM Heap Memory: 192 GB (configured per node)Max Timer Driven Thread Count: 192Processor Count: Over 10,000 processors across the flowsJava version 11We are experiencing the following issues:Frequent node disconnectionsFlow synchronization failures during node reconnectsOccasionally, policies appear empty when nodes rejoinWe have ensured the flow.xml.gz, authorizations.xml, and users.xml files are consistent across all nodes. However, the issues still persist.Could you please advise:What could be causing these frequent node disconnects and flow sync failures?Is there an upper limit on the number of processors or thread count that could lead to instability?Are there recommended JVM GC or NiFi tuning settings for high-core, high-memory environments?Any insights or tuning recommendations would be greatly appreciated.

--1. create a table with hive3create external table aaa( id string comment '') partitioned by (data_dt ); --2. insert hive with dynamic partitioninsert overwrite table partition(data_dt)select id,data_dt from ( select 1 id,'2021' data_dt union all   select 2 id,'2022' data_dt union all   select 3 id,'' data_dt union all )t; --3.check data select id,data_dt,length(data_dt)  len  from aaa; the result:iddata_dtlen1202142202243__HIVE_DEFAULT_PARTITION__26--4.wired results with length(partition )select  *  from aaa where length(data_dt)=26 ; -- return  no data  WHY  ?????😂select * from (select  *, length(data_dt) len from aaa )t where  len=26 ; -- return  no data  WHY ?????🙀select  *  from aaa where data_dt ='__HIVE_DEFAULT_PARTITION__' ;  --return 1  record      

Hi, everyone.We have a three-node Apache NiFi Cluster 1.18.0.Content archive is enabled (15 minutes deep), content repository is on it's own disk.Today we noticed that 2nd node use a lot of disk space by content repository (>80%). I've checked archive files and they fill only about 500 Mb disk space. Other space was used by current content data.Other nodes used content repo as usual -- about 25%.Because we start to receive next errors and 2nd node performance goes down, I've start 2nd node offloading (~ 13:10 on graph down below):Unable to write flowfile content to content repository container repo0 due to archive file size constraints; waiting for archive cleanup. Total number of files currently archived = 63After 2nd node was successfully offloaded I've connect it to the cluster again and now it works fine -- data processing, content repo uses as usual.But when offloading processed, I've track disk space utilization graphs and noticed that 1st and 3rd node did not utilize disk spa

Dear Team,I am trying to register the environment, I am using aws me-central-1 region. i am trying to register the environment . I am getting error on datalake provisioning . i am unable to proceed further on it. please do the needful.Error Message:Datalake creation failed. Data Lake creation failed on 'dl-aws-02' cluster. Reason: Stack status: CREATE_FAILED, cluster status: CREATE_FAILED, reason: Multiple exceptions occurred during resource creation: 1. Error reason: Cannot execute method: runInstances. You have requested more vCPU capacity than your current vCPU limit of 32 allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. (Service: Ec2, Status Code: 400, Request ID: 365d46c8-898d-453e-b612-418ca5c8eaf8) 2. Error reason: Cannot execute method: runInstances. You have requested more vCPU capacity than your current vCPU limit of 32 allows for the instance bucket th

Hello,I have seen multiple sources (for example this press release) state that Cloudera and CrewAI have partnered and it should be easy to deploy a CrewAI project on the Cloudera AI Workbench, but I can't find any documentation about it and no option to do so in the UI.Does anyone know more about it or has maybe even already made a CrewAI project on Cloudera?Best Regards

Recently started getting the following error when attempting and odbc connection using 2.06.09.1009 windows 64-bit odbc driver:"[Cloudera][DriverSupport] (1100) SSL certificate verification failed because the certificate is missing or incorrect. (1100) (SQLDriverConnect); [HY000] [Cloudera][DriverSupport] (1100) SSL certificate verification failed because the certificate is missing or incorrect. (1100)"Upon further investigation using openssl against the cacerts.pem file we discovered there are several components in the pem file that have expired.Where can we get an update cacerts.pem file?Thanks. 

Hi, I was able to setup a secure NiFi cluster with two nodes on two different AWS EC2 linux instances using the certificate generated using NiFi toolkit but I am not able to secure it with company provided certificates. I followed this article - https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-provided-certificates and was able to execute all steps on one of the nodes and tried to access the UI shutting down the other node but still I get below error - <server name> didn't accept your login certificate, or a login certificate may not have been provided. ERR_BAD_SSL_CLIENT_AUTH_CERT Can you please suggest what can be done to address this issue or point to some walkthrough article other than the link I mentioned above where the enterprise provided certificate has been deployed successfully.

I am trying to run an example from the cloudera documentation site in python. I am getting ssl certificate error: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) Getting a certificate is not a solution to my problem. Can I ignore the certificate validation when accessing the api, similar to how it can be done through a regular request? For example: import requestsres = requests.get(url, auth=(login, pass), timeout= 1000000, verify= False)  my code: import cm_clientcm_client.configuration.username = logincm_client.configuration.password = passapi_url = api_host + ':' + port + '/api/' + api_versionapi_client = cm_client.ApiClient(api_url)cluster_api_instance = cm_client.ClustersResourceApi(api_client)  

Hello,I am working on getting registry setup using docker and for some reason I can't seem to get LDAP authentication to work. This is a snippet of the error I am getting when I try to login:2025-07-17 19:24:09,286 DEBUG [NiFi Registry Web Server-28] o.s.s.l.authentication.BindAuthenticator Failed to bind with any user DNs []2025-07-17 19:24:09,447 DEBUG [NiFi Registry Web Server-28] o.a.n.r.s.ldap.LdapIdentityProviderorg.springframework.security.authentication.InternalAuthenticationServiceException: ldap.mydomain.com:686 at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:190) at org.springframework.security.ldap.authentication.AbstractLdapAuthenticationProvider.authenticate(AbstractLdapAuthenticationProvider.java:80) at org.apache.nifi.registry.security.ldap.LdapIdentityProvider.authenticate(LdapIdentityProvider.java:247) at org.apache.nifi.registry.web.api.AccessResource.crea

Dear Team,I have received my developer license . i tried to login , its redirecting  to "To get started, please accept our online services terms" page (Cloudera - Onboarding). here i read the entire terms and continue. i scrolled to bottom page. but "continue" button is not enabled.Note: i have removed my extensions.Please help me on itThanks!

HelloWe're testing the iceberg tables functionality on our CDP 7.3.1 environment, and after reviewing the documentation and performing some tests, it seems that the only way to compact the tables efficiently after modifications is via Spark.Impala's 'optimize table' command rewrites the whole table (the file size threshold option is not implemented till version 4.5), and Hive seems not able to compact them in this version 4 included in our CDP.The Spark procedures rewrite_position_delete_files and rewrite_data_files work correctly, but we wonder if we've misunderstood the documentation or maybe are missing any alternative.Thanks in advance for your help.Regards,Miguel

Hi everyone,Currently trying to create a pySpark application that requires Numpy.I have tried to install it on every worker and master nodes but without success since the notebook always returns the error that it does not exist.Had anyone done this? I believe this should be something rather easy, but my experience is probably not enough to get it at this point.Thanks