Q&A

HiWe are currently operating a three-node cluster using Apache NiFi version 2.1.During a recent deployment of NiFi flows, we encountered an issue related to replication inconsistencies across the nodes.This problem resulted in one of the nodes being out of the cluster and remained in disconnected state.Furthermore, the affected node did not attempt to reconnect to the cluster on its own.Although the node was disconnected from the cluster, it continued to operate independently, allowing it to consume messages, execute database queries, and perform actions as a standalone instance.I would like to know two things hereIs this expected behavior for nifi node to be active although it is not in clusterCan we alter the behavior through configuration to stop processors temporarily while node is not connected to cluster?Thanks for in advance for your inputs.  Let me know if more details are required.Thanks    

Hi all,we are experiencing a progressive socket leak from the LLAP HiveServer2 daemon to HDFS DataNodes in a cluster running HDP 3.1.5The fd grow constantly until the limit of 6400 and the only resolution is to restart hiveserver2Interactive:[hive@myhost hive]$ netstat -tanp | grep "CLOSE_WAIT" | awk '{print $NF}' | awk -F'/' '{print $1}' | sort | uniq -c(Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.)1734 -55443 27915 49167[hive@myhost hive]$ ps -ef | grep 2791hive 2791 1 31 Apr11 ? 6-01:44:32 /usr/jdk64/jdk1.8.0_112/bin/java -Dproc_jar -Dhdp.version=3.1.5.0-152 -Djava.net.preferIPv4Stack=true -Xloggc:/var/log/hive/hiveserverinteractive-gc-%t.log -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/grid/0/tmp/hsi_heapdump.hprof -Dhive.log.dir=/var/log/hive -Dhive.log.f

Hello,I need to enable the HSTS header for Cloudera Management Services - Host Monitor Service and Service Monitor Service.Whilst I managed to do this for all the other services in the cluster using CORE_SETTINGS -> via HTTP Strict Transport Security parameter, this doesn't seem to be applied to Host Monitor Service and Service Monitor Instance from Cloudera Management Services . Is there a way to enable this?I am running a cloudera manager 7.13.1 with a cloudera runtime 7.3.1.Thank you so much!

Hi Cloudera Cummenity,    This documentation lists the versions of components used in Cloudera CDP 7.1.9 SP1:https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/runtime-release-notes/topics/rt-pvc-runtime-component-versions-719sp1.htmlThe Hadoop version is 3.1.1.7.1.9.1000-103.However, when I look at Apache Hadoop Website, I could not find the same version there:The similar thing also happens for HBase:So my question is, why can't I find the hadoop and hbase version that Cloudera CDP used from Apache Hadoop and HBase website?

HiWe are using SiteToSiteReportingRecordSink to generate file transfer report with below settingWe are trying to send the data to the same server where we have added input port processor.Below is the nifi.property setting, do we need to change anything here ?? # Site to Site propertiesnifi.remote.input.host=52564f1f10ffnifi.remote.input.secure=truenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.remote.contents.cache.expiration=30 secs# web properties ############################################### For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.# Be aware that changing these properties may affect how your instance can be accessed without any restriction.# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.nifi.web.http.host=nifi.web.http.port=nifi.web.http.network.interface.default=we are getti

hi How do we get a report for all the file transfers for a given period ?We are using Nifi to only transfer file from one location to another using SFTP and no transformation on file data.Would like to generate a report with the transferred file name and its transfer status.Thanks  

Hello I need helpI want to unzip file.zip with NiFiI have used unpackContent processor but that didn't work because of my nifi version 1.13I have also used executeStreamCommand to execute python script but I didn't found until now the solutionCan you help me Thanks

I am using HBase 2.1.6.Exception happened As following : hbase(main):001:0> scan 'FPCA_ITEMS_TY_NEW',{STARTROW => '08', LIMIT => 2}ROW COLUMN+CELLorg.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive/data/default/FPCA_ITEMS_TY_NEW/bf92b15900f190730a5482b53d350df0/cf/ab741ac0919480a47353778bda55d142202502239bf346dbbfc6475c8967734c2edfaaf4at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:440)at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:354)at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:73)at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6581)at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6745)

Hello everyone,I'm seeking technical advice about designing data flows in Apache NiFi. I extract data from my source (Postgres), and my question is whether I should separate as process groups for Extract, Transform, and Load, or one process group covers end-to-end ETL for one table.

Hello Experts,I am using below end point to send update variable registry request -"/process-groups/<id>/variable-registry/update-requests" and check the status using below url in loop "/process-groups/<id>/variable-registry/update-requests/<variableRegistryUpdateId>".It works well for some time and all of a sudden it starts behaving strange.Status will not get change even after 200 times status api call with almost 2 mins wait total.If we see the status api response , it shows like below - where it is stuck at 'Stopping affected Processors' stage , but if we look at effected processors, in one case there is not effected processors to stop and in another case all processors are already in stopped state but still it is not at progressing.Any idea what could be the reason?Update status check response Case 1 :{"request": {"requestId": "573b0dd9-6f14-4ca2-80d1-513d93451f22","uri": "https://o9nifi-1:9091/nifi-api/process-groups/9e391108-646a-3c3c-a642-a4fbce1d9107/var

Greetings everyone, thanking you in advance for your support. I installed Nifi version 1.16.3 in cluster format, with ZooKeeper apache-zookeeper-3.6.3 separately, on a Linux AWS instance to upgrade from 1.11.4 because I need to work with a more recent API for BigQuery uploads. However, when I complete the process and launch the Nifi service, the processor bar doesn't appear in the UI. The Java version I have is: OpenJDK version "1.8.0_382" OpenJDK Runtime Environment (build 1.8.0_382-b05) OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode). I need some guidance on this. If anyone can help, please.nifi-app.log2025-04-24 19:17:53,449 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@756b2d90{nifi-error,/,file:///home/bxnifi/nifi-1.16.3/work/jetty/nifi-web-error-1.16.3.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.3.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.16.3.war}  

Hello Team, I am looking to control processers from NiFi iteslf. I learn that can be possible with Nifi Rest Api. But i am not able to connect api with InvokeHTTP processor. I am using NIFI 2.0.0-M4There are multiple post/articles available on internet but unfortunately those are showing and talking about old version of these processor.At first  I have tried to access it with StandardSSLContextService2.0.0-M4 by saving certificate and then use it from java keystoreThen i tried to remove SSL certificate and disable all remote securities to push Nifi with "http". Here i tried with StandardOauth2AccessTokenProvider 2.0.0-M4. but it is still showing below error If you could comment on this, it will be appreciated Or at least  Please guide me with some latest tutorials/articles to understand this latest processors. Thank you,

Hello, I'm investigating Nifi toolkit (1.16 if it matters)to add key/value pairs to a param-context programmatically( I can't use existing PC providers )Nifi toolkit documents the following action:#> nifi merge-param-context -helpAdds any parameters that exist in the exported context that don't exist in theexisting context. Overwrites any existing inherited parameter contexts with theprovided list....-i,--input <arg> A local file to read as input contents, or apublic URL to fetchI would like to know if the key/value pairs to add to the PC ( -pcid argument )can come from the contents of the -i fileIf yes, what is expected format of this file please ?( random testing and error messages didn't help much)Regards

Hello, I need help to renew validity of web certificate for Nifi.It is for local host and Nifi2.0.0-M4Please refer below screenshot.Thanks & Regards,Gaurav Mahajan 

When I try to install the CM agent 5.16 on SLES12 SP3 I get following error: m4dn136:~ # zypper install cloudera-manager-daemons cloudera-manager-agentLoading repository data...Reading installed packages...'cloudera-manager-daemons' is already installed.No update candidate for 'cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.sles12.x86_64'. The highest available version is already installed.Resolving package dependencies...Problem: nothing provides python-psycopg2 needed by cloudera-manager-agent-5.16.1-1.cm5161.p0.1.sles12.x86_64 Solution 1: do not install cloudera-manager-agent-5.16.1-1.cm5161.p0.1.sles12.x86_64 Solution 2: break cloudera-manager-agent-5.16.1-1.cm5161.p0.1.sles12.x86_64 by ignoring some of its dependenciesThis package is not part of the SLES12 SP3 distribution. Where can I get it?

We are using ConsumeImap Processor in NiFi. I have schedule the job to run once in the day. When the job is running at scheduled time its reading all the mails but processing only one.But when I am triggering it manually, its reading all the mails and processing all the mails.What could be the issue? below is the configuration for the same  

Hi All,I’ve created a .hql file containing multiple INSERT  statements  for Hive tables and I'm calling the script using Beeline. I need help with capturing error codes in a log file.Also, if any INSERT statement fails, I want the continue the  remaining statements  and need to be write a log into file.I am using below command but it is writing more details log > archive_$(date +\%Y\%m\%d).log 2>&1'Please advise how to handle this? 

Hello All,We have created a custom parcel for Cloudera runtime version 7.1.8 with OS RHEL 7. Our cluster is now operating in mixed OS mode, with some nodes upgraded to RHEL 8 while others are still running on RHEL 7. After the upgrade, Cloudera Manager UI is showing an error for the parcel: "Parcel not available for OS distribution RHEL 7."Is there a way to create parcels for both OS versions so that they can be distributed in mixed OS mode?Thanks,

Hi Community,I'm working on a NiFi setup where I use a dedicated template to track the status of FlowFiles from various other templates. The status of each FlowFile is logged in a specific pattern, and I'm using this pattern to extract and persist status information.Here's a brief overview of the current approach:TailFile Processor reads log entries from a specific log file.SplitText Processor splits the log content line by line.ExtractGrok Processor extracts relevant fields using a defined Grok pattern.ReplaceText Processor restructures the data to a desired format (e.g., JSON).PutDatabaseRecord Processor stores the structured data into a database.Problems Faced:Queue Build-Up & Performance Bottleneck:TailFile often brings in large chunks of data, especially under high log volume.The SplitText processor cannot keep up with the rate of incoming data.This leads to large unprocessed FlowFiles piling up in the queue.FlowFile Explosion & Choking:Once a large FlowFile is split, it r

Hi Team, We are trying to write data from kafka to ozone using nifi but somehow its not working rather its not giving any expection also Flow details generate flow files -> publish kafka -> cosume kafka -> put hdfs (ozone)wherein generate flow files -> put hdfs (ozone) -> works (which confirms there is no authentication issue)changes made in  the core-site.xml config   <property>    <name>fs.defaultFS</name>    <value>ofs://devozone</value>  </property>Are we missing any configs here on the processor level    are we missing anything Need assistance to resolve this issue 

I am trying to create a Ranger plugin audit directory on HBase programmatically using an API. However, I couldn't find any relevant API or documentation that explains how to achieve this.

Hello everyone. I need some help.I am currently working on a Proof of Concept (POC) for building a platform for a client, which is why I am unable to open a case and am asking my question here in this community.We are using spark3 and kudu, and there seems to be a problem. MIT Kerberos is set up, and I'm working in a CM7.10.1, CDP7.1.7sp2 environment.It's a Spark On YARN environment, and I'm executing commands like this: spark-submit --master yarn --deploy-mode cluster \--keytab /etc/security/keytabs/user1.keytab \--principal user1@example.com \--jars /opt/cloudera/parcels/CDH/lib/kudu/kudu-spark3_2.12.jar \....The spark application runs normally on YARN and operates, but an error occurs when the Kerberos ticket is renewed after 7 days. Since the spark application restarts once by default, I think it will be completely killed after 14 days, but for now, it's still running after restarting.Spark errororg.apache.kudu.client.NonRecoverableException: cannot re-acquire authentication t

HiWe are using NiFi API to create ListSFTP processor.When trying to connect to remote host we get below error.Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methodsWhen we update the "Password" manually in the processor it works fine.We validated that the password value is sent with correct data but for some reason its not getting set using API.Do we need to do anything extra for setting the password using API, Thanks 

账号邮箱。之前的被注销了，现在无法接受验证码，无法登录，请问怎么更改账号绑定邮箱

in CDP Public Cloud 7218 RangerRaz Cluster.Below spark job is not generating logs though it Run Successfully.[cloudbreak@ajay7218razdh-master0 ~]$ spark3-submit \> --master yarn \> --deploy-mode cluster \> --conf "spark.hadoop.hadoop.security.authentication=KERBEROS" \> --conf "spark.hadoop.hadoop.security.authorization=true" \> --conf "spark.hadoop.fs.s3a.delegation.token.binding=org.apache.knox.gateway.cloud.idbroker.s3a.IDBDelegationTokenBinding" \> --conf "spark.hadoop.fs.s3a.idb.auth.token.enabled=true" \> --conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider" \> --conf "spark.hadoop.fs.s3a.security.credential.provider.path=jceks://hdfs/user/infa/knox_credentials.jceks" \> --conf "spark.hadoop.fs.s3a.endpoint=s3.amazonaws.com" \> --conf "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \> --conf "spark.driver.extraJavaOptions=-Djavax.net.debug=ssl:handshake" \> --conf "s