Q&A

Since the later versions of NiFi no longer provide the detailed reason an XML FlowFile fails validation against a schema (just a generic Validation Failed), I have been trying to create my own using the ExecuteGroovyScript processor. I am new to NiFI and Groovy, but I am an experienced .Net developer in C#.Below is my Groovy script in the procesor:import groovy.xml.XmlUtilimport javax.xml.transform.stream.StreamSourceimport javax.xml.validation.SchemaFactoryimport javax.xml.XMLConstantsdef flowFile = session.get()if (!flowFile) returndef xmlContent = flowFile.read().getText("UTF-8")def schemaFile = new File("My_XML_Schema.xsd")def schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI)def schema = schemaFactory.newSchema(schemaFile)def validator = schema.newValidator()def validationErrors = new StringBuilder()validator.setErrorHandler(new org.xml.sax.helpers.DefaultHandler() {@Overridevoid warning(org.xml.sax.SAXParseException e) throws org.xml.sax.SAXException {v

Hello.I have configured the HSM.However, testing per Validating Key HSM Settings fails.Settings----------# sudo service keyhsm settingskeyHsm Server Configuration information:keyhsm.management.address : 127.0.0.1keyhsm.server.port : 9090keyhsm.management.port : 9899keyhsm.service.port : 19791keyhsm.jvm.heap.mx.gb : 2keyhsm.hardware : ncipherModule OCS Passwordthales.ocs_password : V6DvCGbrimcD7UPA6QnoOhJb37mFOuoScY30LfWjDwvCybL4E99eT5SKUOcZdu6pq5y66iROKZboNagXzCRxl4x7+N3C3ypKzUJV5UwV3hBjaNS2/qpbyUQD+UUgCoOkm6CxuiOFbOu9CmhnlHBC2UwxqjtnMrtzCR7XMI/Vegm6iZGwR9YWFSeTRRjPkQ/Rhce81hTIqmk7U0+LGHEK+niuARmVt6EG7nmDvZMQufqhOoG2yd4FlYKv2Lv9dDKEKTByv/xoT+/Qh/+Y+8ZbuZHDbEPPzJrq6K848jXhV2wBGTt4RJeKayBzUjwix2LREonTcOctgDf/oJhuIbS2dA==Status--------[root@cloudera-manager ~]# sudo service keyhsm statusKey HSM is running as of Thursday October 23rd, 2025 (10:32 AM), (Started : Thursday October 23rd, 2025 (10:29 AM)) Validation failed --------------------What is port 11371?# curl -k https://loca

Hello.I am on CDP CDH-7.3.1-1.cdh7.3.1.p400.67986116, using Spark 3.5.4.7.3.1.400-100 and no longer able to to find HWC classes under com.hortonworks.spark.sql.hive.llap.*. Is HWC deprecated and/or unsupported on CDP 7.3.1+?Thank you.David.

Unable to start Nifiregistry after Enable Ranger Dependency

I am using NIFI to read data from FTP and push it to HDFS. My business process is to schedule workflows for subsequent computing tasks through dophinschedule. One detail here is that my workflow will execute Alter for corresponding analysis before running.I have noticed that the Spark task occasionally encounters a FileNotFound error during runtime, which may cause the task to fail. It is speculated that this is because partition information has already been added. At this time, when the task is running, NIFI is still writing data to the corresponding partition, and the file being written will cause this error. How to optimize this problem? 

Hi All, I am trying to fetch data from the simple wttr api. The below works fine from curl or postman at the same time: curl http://www.wttr.in/paris?format=j1 But accessing same from invokeHTTP gives me connection refused or connection reset. I have tried by putting higher timeouts and adding user agents, but no resolutionDo I need to open the internet connectivity somewhere or something?.  invokeHTTP Image 

Hi , This is my current NIFI flow where i am consuming data in NIFI using ConsumeKafka_1_0 and publishing using PublishKafka_1_0 . Now HiveMQ (MQTT) → Kafka Topic A → NiFi consumes → NiFi publishes → Kafka Topic BMy requirement is send data in real time , i want to do real time data streams like kafka streams .How to achieve that using NIFI . Is kafka streams available in NIFI ?iam not doing and data transformation or any other operation to data. Iam just simply consuming and publishing data , but now i want to do it in real time .

Hi! I have flows on NIFi and in many places, I have to save data to the database. I mainly use PutSql or PutdatabaseRacord processors. So far, everything was working fine, but today we wanted to test our data and issued 15 messages at once. Unfortunately, in several places, during data saving or updating, an error appeared - error.code 1205 error.message Transaction (Process ID 76) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. error.sql.state 40001 - after which some data was saved or updated correctly, others unfortunately were not. How can I resolve this situation, and how can I protect myself against something like this? 

Hi!I am getting the following error while restarting the Ranger KMS service after configuring the HSM.Please advise.Dalier.Thu Oct 16 15:36:41 EDT 2025JAVA_HOME=/usr/lib/jvm/java-openjdkUsing -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTSUsing /var/run/cloudera-scm-agent/process/1546345804-ranger_kms-RANGER_KMS_SERVER as conf dirUsing scripts/control.sh as process scriptCONF_DIR=/var/run/cloudera-scm-agent/process/1546345804-ranger_kms-RANGER_KMS_SERVERCMF_CONF_DIR=Thu Oct 16 15:36:42 EDT 2025: [I] Using : /opt/cloudera/cm/lib/postgresql-42.5.1.jar and org.postgresql.Driver for postgresql database typeERROR: Hadoop KMS could not be startedREASON: java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.crypto.key.kms.server.KeyAuthorizationKeyProvider$KeyACLs.startReloader()" because "this.implKeyACLs" is nullStacktrace:---------------------------------------------------java.lang.NullPointerException: Cannot invoke "org.apache.hadoop

Hi guys,i've just started to work with parquet files, all is running with database own export logics, but they are not traceable - use NiFi was the Idea. Now im just annoyed how bad i am to handle this ... seems no default export available for this, install extensible-bundles ... NoClass here and there etc...Did anybody of you solved to add Parquet to NiFi 2.5.0?I've downloaded and provided nifi-parquet-nar-2.5.0.nar, nifi-hadoop-nar-2.5.0.nar and nifi-hadoop-libraries-nar-2.5.0.nar still NoClassDefFoundErrors in this order of log (single named)- org/apache/nifi/serialization/RecordSetWriterFactory- org/apache/nifi/processors/hadoop/AbstractFetchHDFSRecord- org/apache/nifi/processors/hadoop/AbstractPutHDFSRecord- org/apache/nifi/serialization/RecordReaderFactory- org/apache/nifi/serialization/RecordSetWriterFactory- org/apache/parquet/io/OutputFile- org/apache/parquet/io/InputFileAnybody who can helpt me?

Hello I need help how to make a flow which can handle 2 pieces of data from user. Let us consider a scenario-> User submits a form having some fields like name, age, address and also a file having something say a list of his hobbies(just for example). How do we process these together and return a response back?I have my flow like this HandleHttpRequest->ExecuteScript (Groovy)-> HandleHttpResponseI am able to do one logic only that either handle the json body say {"name":"Test Name","age":"25","address":"Test address"} or the file which gets sent.How to process both inside my Groovy Script? 

Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for user/myname/.cm/distcp-staging/2025-09-21-05-14-47-c4f9/intermediate.1 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:447) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:152) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:133) at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:3566) at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:3360) at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:3336) at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2899) at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2938) at com.cloudera.enterprise.distcp.util.DistCpUtils.sortListing(DistCpUtils.java:427) at com.cloudera.enterprise.distcp.mapred.StatusReducer.lambda

Hi everyone, I would like some clarification. Thank you.I have a question about accessing the CDP Private Cloud Data Services's web UI and understanding which pods or services are related to the web UI.The URL for the CDP Private Cloud Data Services is:https://console-[namespace].apps.am.ocp.poc/I would like to know how we can determine which pod this web page is running on and under which service?Additionally, if we adopt this method to install the Cloud Data Service:https://docs.cloudera.com/cdp-private-cloud-data-services/1.5.2/installation/topics/cdppvc-installation-steps.htmlIf our OCP is installed in a private network, do you have any suggestions on how to allow the web page to be accessed externally? (e.g., load balancer, ingress, or external gateway)

hi community cloudera,i have a cluster with two Zookeepers.1. Zookeeper is used by all services in the cluster.2. Zookeeper-kfk is used by the Kafka service.however, for Ranger to manage permissions in Kafka, Kafka needs to use the same Zookeeper that Ranger uses.therefore, I need to merge the two Zookeepers. I've tried using zkcopy.py, but I always get the error "ERROR: Source path "/var/lib/zookeeper" does not exist.", but the directory exists in the source.ls -l /var/lib/zookeepertotal 8-rw-r--r-- 1 zookeeper zookeeper 2 Oct 8 11:24 myiddrwxr-xr-x 2 zookeeper zookeeper 4096 Oct 8 11:25 version-2cloudera's official support doesn't have a procedure for this issue. Therefore, I'd like to know if anyone has managed to run zkcopy.py with success or used another method to merge two Zookeepers in the same cluster?

Hi Team, I am unable to impala in the nifi executor, as i have to use impala using a jdbc connection by placing the impalajdbc41.jar in a customized location, and when restarted the nifi the jython script worked like a charm, however when using the nifi executor it says java class path not found, why is this causing the issue. Any leads would be appreciated RegardsKhursheed

Hello,We have problems with parcels in our PROD cluster (CDP Runtime 7.1.7), we tried to remove old parcels (CDH 6.3.3, CFM 1.0.1.0, CFM 1.1.0.0) on host dallas20- After the manual delete, Cloudera Manager still shows some parcels with status Distributed.- The parcel folders on disk are not correct now, probably because of the manual cleanup.- Also, the cluster is slow and some services presents an error like this in Parcels tab on CM:Error when distributing to dallas20.metromadrid.net: [Errno 20] Not a directory: /opt/cloudera/parcels/STREAMS_MESSAGING_MANAGER-2.1.0.2.0.1.0-2/meta/parcel.jsonWe need help with these points:How can we clean the old parcels in a safe way so CM shows the correct status?How to fix the error [Errno 20 Not a directory] and make parcel distribution work again?Is there a supported procedure to reset parcel states in Cloudera Manager?This problem is blocking us to free space in /opt/cloudera/parcels/ and manage parcels normally.We are using CDP 7.1.7 (whic

Hello Team,Currently we use NiFi’s InvokeHTTP to send files to a MiNiFi IoT device. The MiNiFi listenhttp is configured in the YAML and we are calling the device with:http + ip (device)+portSecurity requires we move this to HTTPS. On the NiFi side we configured SSLServiceContext and added a truststore. On the MiNiFi device we configured listenhttp in its YAML and provided a keystore and truststore (certificates are not machine-signed). After switching to HTTPS we see a hostname verification error Question: is there a supported option in Cloudera / NiFi’s InvokeHTTP to use only a truststore and skip hostname verification? Alternatively, is there a recommended way to bypass or relax hostname verification on the MiNiFi side so the call succeeds? Thanks

Hi AllUnder CDP 7.3.1, our Impala's log is flooding with messages likeORC read request to already read range. Falling back to readRandom. offset: 16248424 length: 105200 colrange_offset: 15988393 colrange_length: 365231 colrange_pos: 16353624 typeId: 94 kind: data filename: .......from queries reading ACID ORC tables created from Hive.We've been searching about that and the only thing we've found is the code that triggers that message.void HdfsOrcScanner::ScanRangeInputStream::read(void* buf, uint64_t length, uint64_t offset) { Status status; if (scanner_->IsInFooterRange(offset, length)) { status = scanner_->ReadFooterStream(buf, length, offset); } else { ColumnRange* columnRange = scanner_->FindColumnRange(length, offset); if (columnRange == nullptr) { status = readRandom(buf, length, offset); } else if (offset < columnRange->current_position_) { VLOG_QUERY << Substitute( "ORC read request to already read range. Falling back t

We are trying to optimize cluster provisioning by preloading parcels into our VM images. The goal is to have parcels already available on each host before they are added to the cluster, so we can avoid re-distribution during host onboarding.Here is what we tested and observed:We baked parcels into the VM image under /opt/cloudera/parcels/.When we add a host created from this image into the cluster, Cloudera Manager still triggers parcel distribution.During this process, CM deletes the preloaded parcels on the new host and redistributes them again.We have tested the following without success:Setting Auto Distribution = falseVerifying parcel existence manuallyCreating/updating the .distributed_parcels file with the correct versionsOur conclusion so far is that Cloudera Manager does not trust preloaded parcels, but instead enforces consistency against its internal parcel state database, which causes redistribution and deletion even if parcels are already present.Our question:Is there any

I an trying to configuring NiFi regoistry (2.5.0) with LDAP, have successfully configured so I can login using AD credential. However I am not able to see users or groups from AD in the admininstration sections.I have checked the user  and group sections authorizers.xml) are the same as NiFI, as I can see users and groups when I go to Users or Policies section.I am not using LDAPS (use LDAP in the dev envirobment), and this works fine in NiFi 2.5, and just cant see why it doesnt work with NiFI registry using teh same LDAP usergroup provider settings. Any thoughts?

Hello there.. I am using replacetext processor where i am using the below query as my replacement value.INSERT into log_hive.log_ozone_retention_jobs(Table_Name,Process_Date, Process ,Status ,Source_Count ,Target_Count ,Start_Time ,End_Time ,Notes)select 'Customer_Voip',${date_list}, 'Insert_Into_Ozone','Insert Successful',${source_count},null,'${start_time}',current_timestamp(),null;date_list is an attribute value coming along with the incoming flowfile.. My downstream processor is PutHiveQL where it executes the insert query provided above..  I was successfully able to insert a single record for that day using the  PutHiveQL processor.. However, today i found that PutHiveQL actually inserted 2 records for the same day as below... the incoming flowfile had date_list value as 20250401 for example and is expected to insert only 1 record but inserted 2 records with 2 different end time... this happened with all the dates that were meant to be loaded...P

We are using NiFi’s InvokeHTTP to POST files over HTTPS to MiNiFi IoT devices. MiNiFi is configured with JKS keystore/truststore files containing certificates signed by our CA and bound to the device hostnames.NiFi InvokeHTTP fails with a hostname verification error when connecting to the devices. We need to either disable the hostname the Could you please advise the recommended else how should configure invokehttp in nifi should we also add import certif minifi to nifi trustore please what it the logic to activate the https and send files to minifi

I manage NiFi installations for various clients within our company. On any servers or client systems running a Windows-based operating system, and on any version of NiFi I currently maintain (1.11.4, 1.16.3, 1.19.1), NiFi does not roll over logs using the default logback.xml. Example:  <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

Looking for help with Cloudera Data Platform Spark job optimization. We're running large-scale ETL jobs that are timing out during the shuffle phase, consuming excessive cluster resources and causing memory spillage.The jobs process ~500GB datasets but execution times have increased 3x after migrating to CDP. Need someone experienced with Spark tuning on Cloudera and YARN resource management to identify bottlenecks.Seeking 3-4 hours remote performance analysis to optimize job configuration and cluster settings. Must be resolved by Tuesday for our data pipeline SLA.

I was looking for a server sent event (SSE) client in Apache NiFi, however, I couldn't find any ready processor that can do that.  I started implementing an SSE client using Python script and used ExecuteCommand processor to run this script. However, the script needs to be terminated in order to send the processor output to the next step through STDOUT. (ie. I can't use an infinite Loop "While True:" to listen to the SSE server and output the consumed events on stream). Is there any ideas to implement the SSE client in NiFi such that consumed events are processed one by one to the next processors in real-time?