Ask the Community

hi community cloudera,i have a cluster with two Zookeepers.1. Zookeeper is used by all services in the cluster.2. Zookeeper-kfk is used by the Kafka service.however, for Ranger to manage permissions in Kafka, Kafka needs to use the same Zookeeper that Ranger uses.therefore, I need to merge the two Zookeepers. I've tried using zkcopy.py, but I always get the error "ERROR: Source path "/var/lib/zookeeper" does not exist.", but the directory exists in the source.ls -l /var/lib/zookeepertotal 8-rw-r--r-- 1 zookeeper zookeeper 2 Oct 8 11:24 myiddrwxr-xr-x 2 zookeeper zookeeper 4096 Oct 8 11:25 version-2cloudera's official support doesn't have a procedure for this issue. Therefore, I'd like to know if anyone has managed to run zkcopy.py with success or used another method to merge two Zookeepers in the same cluster?

Hi Team, I am unable to impala in the nifi executor, as i have to use impala using a jdbc connection by placing the impalajdbc41.jar in a customized location, and when restarted the nifi the jython script worked like a charm, however when using the nifi executor it says java class path not found, why is this causing the issue. Any leads would be appreciated RegardsKhursheed

Hello,We have problems with parcels in our PROD cluster (CDP Runtime 7.1.7), we tried to remove old parcels (CDH 6.3.3, CFM 1.0.1.0, CFM 1.1.0.0) on host dallas20- After the manual delete, Cloudera Manager still shows some parcels with status Distributed.- The parcel folders on disk are not correct now, probably because of the manual cleanup.- Also, the cluster is slow and some services presents an error like this in Parcels tab on CM:Error when distributing to dallas20.metromadrid.net: [Errno 20] Not a directory: /opt/cloudera/parcels/STREAMS_MESSAGING_MANAGER-2.1.0.2.0.1.0-2/meta/parcel.jsonWe need help with these points:How can we clean the old parcels in a safe way so CM shows the correct status?How to fix the error [Errno 20 Not a directory] and make parcel distribution work again?Is there a supported procedure to reset parcel states in Cloudera Manager?This problem is blocking us to free space in /opt/cloudera/parcels/ and manage parcels normally.We are using CDP 7.1.7 (whic

Hello Team,Currently we use NiFi’s InvokeHTTP to send files to a MiNiFi IoT device. The MiNiFi listenhttp is configured in the YAML and we are calling the device with:http + ip (device)+portSecurity requires we move this to HTTPS. On the NiFi side we configured SSLServiceContext and added a truststore. On the MiNiFi device we configured listenhttp in its YAML and provided a keystore and truststore (certificates are not machine-signed). After switching to HTTPS we see a hostname verification error Question: is there a supported option in Cloudera / NiFi’s InvokeHTTP to use only a truststore and skip hostname verification? Alternatively, is there a recommended way to bypass or relax hostname verification on the MiNiFi side so the call succeeds? Thanks

Hi AllUnder CDP 7.3.1, our Impala's log is flooding with messages likeORC read request to already read range. Falling back to readRandom. offset: 16248424 length: 105200 colrange_offset: 15988393 colrange_length: 365231 colrange_pos: 16353624 typeId: 94 kind: data filename: .......from queries reading ACID ORC tables created from Hive.We've been searching about that and the only thing we've found is the code that triggers that message.void HdfsOrcScanner::ScanRangeInputStream::read(void* buf, uint64_t length, uint64_t offset) { Status status; if (scanner_->IsInFooterRange(offset, length)) { status = scanner_->ReadFooterStream(buf, length, offset); } else { ColumnRange* columnRange = scanner_->FindColumnRange(length, offset); if (columnRange == nullptr) { status = readRandom(buf, length, offset); } else if (offset < columnRange->current_position_) { VLOG_QUERY << Substitute( "ORC read request to already read range. Falling back t

We are trying to optimize cluster provisioning by preloading parcels into our VM images. The goal is to have parcels already available on each host before they are added to the cluster, so we can avoid re-distribution during host onboarding.Here is what we tested and observed:We baked parcels into the VM image under /opt/cloudera/parcels/.When we add a host created from this image into the cluster, Cloudera Manager still triggers parcel distribution.During this process, CM deletes the preloaded parcels on the new host and redistributes them again.We have tested the following without success:Setting Auto Distribution = falseVerifying parcel existence manuallyCreating/updating the .distributed_parcels file with the correct versionsOur conclusion so far is that Cloudera Manager does not trust preloaded parcels, but instead enforces consistency against its internal parcel state database, which causes redistribution and deletion even if parcels are already present.Our question:Is there any

I an trying to configuring NiFi regoistry (2.5.0) with LDAP, have successfully configured so I can login using AD credential. However I am not able to see users or groups from AD in the admininstration sections.I have checked the user  and group sections authorizers.xml) are the same as NiFI, as I can see users and groups when I go to Users or Policies section.I am not using LDAPS (use LDAP in the dev envirobment), and this works fine in NiFi 2.5, and just cant see why it doesnt work with NiFI registry using teh same LDAP usergroup provider settings. Any thoughts?

Hello there.. I am using replacetext processor where i am using the below query as my replacement value.INSERT into log_hive.log_ozone_retention_jobs(Table_Name,Process_Date, Process ,Status ,Source_Count ,Target_Count ,Start_Time ,End_Time ,Notes)select 'Customer_Voip',${date_list}, 'Insert_Into_Ozone','Insert Successful',${source_count},null,'${start_time}',current_timestamp(),null;date_list is an attribute value coming along with the incoming flowfile.. My downstream processor is PutHiveQL where it executes the insert query provided above..  I was successfully able to insert a single record for that day using the  PutHiveQL processor.. However, today i found that PutHiveQL actually inserted 2 records for the same day as below... the incoming flowfile had date_list value as 20250401 for example and is expected to insert only 1 record but inserted 2 records with 2 different end time... this happened with all the dates that were meant to be loaded...P

We are using NiFi’s InvokeHTTP to POST files over HTTPS to MiNiFi IoT devices. MiNiFi is configured with JKS keystore/truststore files containing certificates signed by our CA and bound to the device hostnames.NiFi InvokeHTTP fails with a hostname verification error when connecting to the devices. We need to either disable the hostname the Could you please advise the recommended else how should configure invokehttp in nifi should we also add import certif minifi to nifi trustore please what it the logic to activate the https and send files to minifi

I manage NiFi installations for various clients within our company. On any servers or client systems running a Windows-based operating system, and on any version of NiFi I currently maintain (1.11.4, 1.16.3, 1.19.1), NiFi does not roll over logs using the default logback.xml. Example:  <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

Looking for help with Cloudera Data Platform Spark job optimization. We're running large-scale ETL jobs that are timing out during the shuffle phase, consuming excessive cluster resources and causing memory spillage.The jobs process ~500GB datasets but execution times have increased 3x after migrating to CDP. Need someone experienced with Spark tuning on Cloudera and YARN resource management to identify bottlenecks.Seeking 3-4 hours remote performance analysis to optimize job configuration and cluster settings. Must be resolved by Tuesday for our data pipeline SLA.

I was looking for a server sent event (SSE) client in Apache NiFi, however, I couldn't find any ready processor that can do that.  I started implementing an SSE client using Python script and used ExecuteCommand processor to run this script. However, the script needs to be terminated in order to send the processor output to the next step through STDOUT. (ie. I can't use an infinite Loop "While True:" to listen to the SSE server and output the consumed events on stream). Is there any ideas to implement the SSE client in NiFi such that consumed events are processed one by one to the next processors in real-time?

I have created a mysql database (scm) & user (scm@localhost) on a RHEL7 server. CM server version is cloudera-manager-server-7.6.7-36957245.el7.x86_64; mysql version is mysql Ver 8.0.42 for Linux on x86_64 (MySQL Community Server - GPL). DB user has following grants...mysql> grant select, insert, update, delete, create, drop on scm.* to 'scm'@'localhost';When the scm_prepare_database.sh script is executed, it returns "All done, your SCM database is configured correctly!"  HOWEVER, multiple database tables are NOT created & the cloudera-scm-server.service will NOT stay started.cloudera-scm-server.log returns multiple errors likeCaused by: java.sql.SQLSyntaxErrorException: Table 'scm.CM_VERSION' doesn't existCaused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSetCaused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSetCaused by: java.lang.RuntimeException: Unable t

i have a JSON payload which contains latest and old orders. The data should be compared and output New, update, deleted. How can do it do that in NIFI?.  "latest_orders": [    { "order_id": "1001", "customer": "Alice", "amount": 250 },    { "order_id": "1002", "customer": "Bob", "amount": 300 },    { "order_id": "1004", "customer": "Diana", "amount": 150 }  ],  "old_orders": [    { "order_id": "1001", "customer": "Alice", "amount": 200 },    { "order_id": "1002", "customer": "Bob", "amount": 300 },    { "order_id": "1003", "customer": "Charlie", "amount": 400 }  ]}Output:    { "order_id": "1001", "customer": "Alice", "amount": 250, "Action":"UPDATE" },    { "order_id": "1002", "customer": "Bob", "amount": 300 ,"Action":"UPDATE"},    { "order_id": "1004", "customer": "Diana", "amount": 150,"Action":"NEW" },    { "o

Hi team,I am currently facing issue connecting to Hive database using python libraries (pyodbc, pyhive, impyla), the Cloudera driver and Apache Knox. While the network level connection to the relevant host and port is successful. (verified via ncat, telnet), I am unable to complete the connection and authenticate through these tools. Connection details:Driver=/opt/cloudera/hiveodbc/lib/64/libclouderahiveodbc64.so;""Host=nonprodm01.cdslindia.com;""Port=8443;""Schema=default;""TransportMode=http;""HTTPPath=/gateway/default/hive""SSL=0;"  # Enable SSL if Knox uses HTTPS"AuthMech=3;"  # For username/password authentication; use 1 for no auth or 2 for Kerberos"UID=username;""PWD=password;"Knox gateway is used to access HiveServer2Error message: Connection failed: "('HY000', '[HY000] [Cloudera][DriverSupport] (1110) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct and confirm if SSL should be enabled for the c

Hello!Following the advice from this post: https://community.cloudera.com/t5/Support-Questions/Accessing-NIFI-Metrics-endpoint-for-Prometheus-without/m-p/399710#M250535I'm trying to set up certificate authentication/authorization from prometheus server to nifi, but getting 403 Forbidden. In Nifi user log i can see that authentication CN of certificate is successful, further  comes the 403 error:Prometheus scrape config: Can you please tell what is wrong here?

We've encountered an issue within Cloudera Object Store (powered by Apache Ozone) (1.4.0 and 1.4.1) which we believe are related to HDDS-9967A fix for this particular issue has now been backported to the 'apache:ozone-1.4' project as part of pull request https://github.com/apache/ozone/pull/9066How do we update the Cloudera Runtime or Ozone parcel version (1.4.0.7.1.9.1054-4) to include this fix? Upgrading to the latest 1.4.2 would be ideal.Thanks Stephen

Hi,We have a NiFi cluster which spans two physical locations, Zone A and Zone B.  We are clustering across locations for ease of management.  In each location we have hosts running NiFi acting as data collectors, enriching the data and then sending to the main cluster via site-2-site (using remote processing groups).My question is:We want data originating in Zone 1 to only hit the NiFi Cluster nodes in Zone 1.  Using site-2-site, even if we only configure the Zone 1 nodes in the RPG, NiFi will report the presence of the Zone 2 nodes hence data originating from Zone 1 will cross the WAN and end up in Zone 2.  Does NiFi have any concept of zones?We currently achieve this by configuring a firewall rule to block outgoing traffic to Zone 1 nodes from Zone 2 (and vice-versa), but this produces warnings in the RPG.Any thoughts or ideas would be appreciated.  Thanks 

Hi Team, We have the following issue, we are using nifi work flow, in which we are trying to execute impala, before implementing nifi  execute script, we tested the script by using jython to test the feasibility of the connections we established is successful and able to get the count in impala, however when using the same connection configuration in nifi execute script processor, we get the java class path not found, the impala jdbc connection is placed in a customized location, impalajdbc41.jar file is not a nar, file, and the customized location is readable, as confirmed by the support and if placed in the nar file location will Please find the logs for reference as an attachment.2025-09-29 15:31:08,495 ERROR org.apache.nifi.processors.script.ExecuteScript: ExecuteScript[id=49498cfb-70fc-1ebf-ffff-fffff0fa2dc8] Processing failedorg.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: java.lang.ClassNotFoundException: java.lang.ClassNotFoundException:

Hello everyone,Is it possible to limit the memory for executing queries for an LDAP group in CDP 7.1.7? This would allow creating a memory pool for each group so that if a user in that group launches a memory-intensive query, it does not impact other users in the cluster.BR!

Hitting a weird issue in Nifi that I can't get my head around.I am consuming AVRO records from Kafka with a timestamp field. The definition is { "name" : "timestamp", "type" : { "type" : "long", "logicalType" : "timestamp-millis" } },If I use a JSON record writer then in the queue viewer I can see the field contains the original long value. However if I use an AVRO writer the field is formatted as a date/time value, e.g. 2025-09-26T15:29:44.466Z.I assumed this was just pretty-printing in the queue viewer. But if I then send those flowfiles to a ConvertRecord process with an AVRO reader and a FreeText writer the timestamp field is now displayed as YYYY-mm-DD HH:MM:SS. Which means I've lost precision and I need to use expression language to re-parse the field to convert it back to a long value.So, I'm forced to us JSON as the intermediary format. If I use AVRO I loose precision.This is Nifi 2.5.

Hi,I read from the release note that the encrypt-config.sh tool from nifi-toolkit has been removed in NiFi 2.0 due to various considerations(https://issues.apache.org/jira/browse/NIFI-13414). I am wondering is there any alternatives to this tool? We heavily relied on this encrypt-config.sh tool to encrypt plain-text passwords in nifi.properties, like nifi.sensitive.props.key, nifi.security.keyPasswd. Now with this tool gone, do we just have to accept the fact that plain-text passwords will remain plain-text?Thanks in advance!!

Hello I have a flow where a queue is having say 250 entries but I am able to see only 100. I don't see any option to see the next sets of entries. How to view them as I can see the count to be increasing in queue but it displays oldest 100.

Hello I have configured certificate based authorization by using manager-authorizer in nifi.properties and then adding property name ="Initial Admin Identity" in <userGroupProvider> and <accessPolicyProvider> sections of authorizers.xmlNow once I do that and import admin cert (which I specified in authorizers.xml) then I can access NiFi and I can bring comonents/processors from Component Toolbar. I can also add users from the Global Menu "Users" section or Add Policies.Issue I am experiencing is that the user I added can only view the processors groups and can't modify them. Apart from this the whole component toolbar is grayed out so not accessible. I went to my admin account and added that user in some Policies but this did not help.I saw in some article that we need to right click on certain processor group and then go to "Manage Access Policies" but surprisingly for my admin account the Add User icon is gray and not able to click it. I have added two photos one sho

Hello I want to know one thing. Let's say we have a flow having a Handlehttprequest then a ExecuteSql processor and then a HandleHttpResponse for example. Daily we receive say 1000 calls to this flow. So where does flowfiles get stored? I see in provenance many many flowfiles. So do they purge or all get accumulated. I have less space so wanted to know.