Q&A

HI @MattWho client asked us to use the common username and password kind of authority, instead of certificate based, so i re-created the  NIFI components   to private VM's NIFI nodes, using the https load balancer,Burt now getting the different error: 2024-10-22 15:46:17,574 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: 53.13.138.70:84432024-10-22 15:46:20,785 WARN [Heartbeat Monitor Thread-1] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine leader for role 'Cluster Coordinator'; returning nullorg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinatorat org.apache.zookeeper.KeeperException.create(KeeperException.java:101)at org.apache.zookeeper.KeeperException.create(KeeperException.java:53)at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2480)at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:

Hello!The general task is to transform xml to csvFor this aim I created following pipeline:       Source xml content is:     <Email E-mail="AR-BIK@GMAIL.COM"> <Data GRN="3214600023849" Date="2024-07-28"></Data></Email>Parameters of EvaluateXPath are:        ReplaceText parameters:      Attribute grn successfully produced with expected value:    But value "3214600023849" is not translated to csv. Only ; delimiter. Encoding is UTF-8 What am I doing wrong?

Hello,Trying to configure Datanode balancer network bandwidth to 4 GB since we have a good 25 GBPS network dedicated to the cluster. However, when set via Cloudera Manager, I get an alert stating max allowed in 1 GB.Is it something hard coded and cannot be increased beyond 1 GB via CM? OR there is a different way to do that? Thankssnm1523

I am facing issue ERROR: Toplevel exception SPAWN status error.Hi,I'm trying to set up the Cloudera data platform private cloud base in my VirtualBox with ubuntu 20.04 for trial purposes.Virtualbox specs:4GB RAM80GB DiskUbuntu partition: /opt - 30GB   /var - 30G      /usr - 7GB      / 17GBas per Cloudera manager server specificationsI have installed the Cloudera Manager 7.4.4and uploaded the Cloudera license to install the latest CDH 7.1.7 for POC purposes. hostname: master/etc/hosts file:127.0.0.1 localhost 10.0.2.15 masterNOTE: While specifying the hosts in cloudera manager. I choose localhost to connect. (scanned) wasn't able to go with master. I am installing these steps in ubuntu virtualbox.   Installation script completed successfully.all doneclosing logging file descriptor>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Adding more caps: {'has_memory':True, 'default_memory_limit_in_

When using the InvokeHttp process for all 4xx HTTP status codes, we receive the HTTP response in the request flow file attribute 'invokehttp.response.body'. However, the attribute holds only 256 characters, and any response bigger than that causes truncation.I would like to hold the entire response body , even it is beyond 256 chars for 4xx status code.

Hi ,I have NIFI cluster with three nodes .Each node has 16 CPU and 100G ram .Currently, the JVM heap memory on the primary node gradually grows until it reaches full RAM capacity and crashes on OOM. Can you please help me to address this issue .Why does the memory on the primary node keep growing ? Is the GC not supposed to clean the heap ?  bootstrap.conf # Java command to use when running NiFijava=java# Username to use when running NiFi. This value will be ignored on Windows.run.as=nifi# Preserve shell environment while runnning as "run.as" userpreserve.environment=false# Configure where NiFi's lib and conf directories livelib.dir=/opt/nifi/libconf.dir=/opt/nifi/conf# How long to wait after telling NiFi to shutdown before explicitly killing the Processgraceful.shutdown.seconds=20# Disable JSR 199 so that we can use JSP's without running a JDKjava.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true# JVM memory settingsjava.arg.2=-Xms40gjava.arg.3=-Xmx40g# En

Hi Team, After enabling ranger in Kafka, I am not able to bring up the Kafka broker instance. The error is:[kafka-raft-io-thread]: Encountered fatal fault: Unexpected error in raft IO threadorg.apache.kafka.common.errors.ClusterAuthorizationException: Received cluster authorization error in response InboundResponse(correlationId=0, data=FetchResponseData(throttleTimeMs=0, errorCode=31, sessionId=0, responses=[]), sourceId=155) Please help resolve this. Thanks in advance.

HiIam new to Nifi and trying to get this resolved.we have a S3 bucket with below structure. and using below pattern to get the file recursively.Though this works if the files are there in home directory of the bucket.But it doesn't work for folder or subfolder.Please let me know how to get the files recursively.Note: Also the same structure has to be created in local with PutFile, which I think is possible once i get the files from FetchS3Object.  

I got this error when register environment of CDP Base on OCP. I don't know where to find the specific log of this error, it just says "Unable to connect to Cloudera Manager!"Can anyone please help me?   

Hello Team,We are looking for a solution or steps to implementing of impala jdbc connections in dbeaver through Kerberos. No LoginModules configured for ClientError log;ep 19 13:02:51.421 ERROR 59 com.cloudera.impala.exceptions.ExceptionConverter.toSQLException: [Cloudera][ImpalaJDBCDriver](500166) Error creating login context using JAAS configuration: No LoginModules configured for Client.java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500166) Error creating login context using JAAS configuration: No LoginModules configured for Client. 

Hello Team,can we import/export hive table to s3 location or nfs share ? as per what i searched it is only supported for hdfs location. can you please help me to know if we can do it and if yes how can we achieve it ?

Hello,Could someone please help me with this caseI want to copy data from an Oracle table to another table that has the same schema. I want to count number of record read in and number of records succesfully inserted to the target, and put those counts into a tracking table. My idea is as belowIn which, the left branch I count the records in, the right branch I insert data to target table and then count records.-QueryRecord: count number of record. Output from left and right branch are the same and as below-ExtractText: to extract just the count to record_in and record_out. The output attribute is as below-MergeContent: Merge these 2 attributes before put SQL. Result and output attributes are as below -UpdateAttribute: to escape user-based inputMy query is something like:INSERT INTO TABLE_1 (TABLE_NAME, RECORD_IN, RECORD_OUT) VALUES ('manually_input', ?, ?)I got the error: ORA-00911: invalid character SQL = INSERT INTO TABLE_1 (TABLE_NAME, RECORD_IN, RECORD_OUT) VALUES ('manually_

Hello, I'm new to NiFi and I'm experiencing problems during the installation phase. Unfortunately, I can't connect to the NiFi web interface. I've scoured the internet, but I still can't resolve the issue. I'm trying to install NiFi version 2.0.0-M4, and when I try to access the web interface by typing https://localhost:8443/nifi in the browser, I continuously get the error message "Unable to communicate with NiFi."Some forums suggest waiting for a bit to allow NiFi to fully start and then trying again, but that hasn't solved the problem either. I also tried using HTTP instead of HTTPS, but that didn't work either. There are no error messages in the nifi-app.log file, and everything seems to be running normally.The strange part is that I successfully followed the same installation steps on my colleague's computer, and it worked fine. However, when I follow the exact same steps on my own computer, it doesn't work. As for dependencies, I have both Java 8 and Java 23 installed on my machi

HI @MattWhoWe have configured the nifi  two nodes cluster on Azure environment with the below nifi properties, we are using private ip address of NIFI, currently the NIFI application is running on ubuntu system without any error, But could not able to access it from web url, can you please guide us what are the steps we need to follow in order to access these from web uRL.we tried through load balancer, The problem in the load balancer is that we are not able to get the Vm's traffic to reach the load balancer.Also followed the configuration steps from the below blogApache NiFi WalkthroughsNifi Properties. # Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may

Hello All,I'm running a Nifi cluster with 3 nodes.  Below are some questions in regards to backup. 1. What's the best way to backup Nifi without stop \ starting?2. Is there a way to backup all the processes running using a backup method in the event I need to restore 1 process I don't have to restore all the processes available?Thanks!

Hi,Does anyone know if we can use glue catalog as metastore for PutIceberg Processor in NiFIMy use case is we need to stream records in AWS s3 in iceberg format from NiFi and we use glue catalog as metastore.Would Really appreciate the help

Hi Community,I have recently upgraded our Cloudera Manager from CM 7.6.7 to CM 7.11.3.  After the upgrade, we have not been able to successfully restart many of our services, which include, hue, hdfs, and hbase.We are currently on CDP 7.1.7 SP2. We intend to upgrade the CDP 7.1.7 SP2 to CDP 7.1.7 SP3 after the issues are resolved.As a pre-requisite, we have installed Python3.8 (Installation was via anaconda) and created an alternatives pointing python3.8 to the custom installation location.So, we can type python3.8 --version and get the right result (Python 3.8.18).After the upgrade, we tried to restart a hue service but get the following exception.   [07/Oct/2024 16:58:09 +0100] 10345 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/hue/hue.sh runcpserver[07/Oct/2024 16:58:09 +0100] 10345 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 10620[07/Oct/2024 16:58:10 +0100] 10621 MainThread re

   Hello!  The configuration of my SplitText is:The task is to split one csv file:  id;description "1234";"The latitude is 12324.24" "2345";"12324.24 this value"on 2 files:  id;description "1234";"The latitude is 12324.24"and id;description "2345";12324.24 this value"But it returns 10000 and more duplicated files!What am I doing wrong? 

Hi, here is a summary of our blocking situation on our CDH Express 6.3.1 installation:- A crash happened on the storage where all the virtual nodes composing our Cluster insist- After solving the issue on the storage, our Cloudera cluster started again working, but with a critical error on HDFS Service Nodename- In fact, we have High Availability configured on 2 nodes, and exploring the situation we saw that Namenode 2 was active and Namenode 1 was DOWN- But the cluster worked- In an attempt to fix the issue, we tried several times to manually start the Namenode 1 from "HDFS --> Instances --> federation --> Namenode1 --> Start" but it was impossible to start manually- So, we decided to disable High Availability but the new configuration has not completed and actually is in pending status- It's in fact impossible to start HDFS service at all because an error pops up saying that "Nameservice nameservice1 has no SecondaryNameNode or High-Availability partner"It is interesting

hi community cloudera, i need to read a Hive table and write the result to an Oracle tablethis flow needs to be executed every hour to update the data in the Oracle tabledoes anyone have a test flow for this example?

I renamed a Hive partitioned table DB1.Tab1 to DB1.Tab1_old and then moved another table with the same name but from another database DB2.Tab1 to the first database DB1.Tab1, using the same rename command. It looks like the partitioned directories were merged because when I deleted the first renamed one (DB1.Tab1_old), the moved table partitions (DB1.Tab1) were deleted. I would like to understand why this happened. Regards

rebooting the machine. I couldn’t find any information about another solution for this problem.Good afternoon. when starting apache nifi 2.0.0 M2 (as well as M1, M3, M4), after a while it closes with an error, the java_home variable is defined, nifi_properties indicated its IP. please tell me what the problem is and a possible solution nifi-bootstrap.log2024-08-13 19:59:22,516 WARN [main] o.a.n.b.process.RuntimeValidatorExecutor Runtime Configuration [AvailableLocalPorts] validation failed: Local Ports [28231] less than recommended [55000] according to [/proc/sys/net/ipv4/ip_local_port_range]2024-08-13 19:59:22,519 WARN [main] o.a.n.b.process.RuntimeValidatorExecutor Runtime Configuration [Swappiness] validation failed: Swappiness [60] more than recommended [0] according to [/proc/sys/vm/swappiness]2024-08-13 19:59:23,442 INFO [main] org.apache.nifi.bootstrap.RunNiFi {javaVersion=21.0.4, availableProcessors=3, maxOpenFileDescriptors=1048576, totalPhysicalMemoryMB=5921, minimumHeap

I have a CML project using a JupyterLab Runtime with Python 3.10 and I want to start a Spark cluster with my CDP Datalake. I'm using the predefined Spark Data Lake Connection in CML which looks like this:```import cml.data_v1 as cmldata# Sample in-code customization of spark configurations#from pyspark import SparkContext#SparkContext.setSystemProperty('spark.executor.cores', '1')#SparkContext.setSystemProperty('spark.executor.memory', '2g')CONNECTION_NAME = "hiaa-dl"conn = cmldata.get_connection(CONNECTION_NAME)spark = conn.get_spark_session()# Sample usage to run query through sparkEXAMPLE_SQL_QUERY = "show databases"spark.sql(EXAMPLE_SQL_QUERY).show()``` When I execute this I get the error:IllegalArgumentException: The value of property spark.app.name must not be nullI'm using the predefined spark-defaults.conf which looks like this:```spark.executor.memory=1gspark.executor.cores=1spark.yarn.access.hadoopFileSystems=abfs://[container]@[storage-account].dfs.core.windows.net```Is

Hello all, We have a couple of flows already writting values into Kafka using Avro format for the value, via an AvroRecordSetWriter that also uses a Confluent Schema Registry. The thing is that now we would also need to write the Kafka key also in Avro format. Is there any easy way to do it, or how would we isolate the key, to use an Avro writer to write it in Avro and then merge it will the original flow file as an attribute to be used as key by the PublishKafka processor? Thank you.Best regards,Miguel

Hi Everyone,I am facing a problem that I try to insert data into hiverserver2 by spark thrift server (actually I use beeline), the job of insert is stucked.I have checked that spark MasterApplication UI page, and find that it shows as following figure.The log of spark thrift server is as following :24/10/16 15:21:39 INFO SparkExecuteStatementOperation: Submitting query 'insert into test_database.test_table (a,b) values (2,33)' with a75190ac-d536-4ee1-a1ff-da42a195a40b24/10/16 15:21:39 INFO SparkExecuteStatementOperation: Running query with a75190ac-d536-4ee1-a1ff-da42a195a40b24/10/16 15:21:40 INFO FileUtils: Creating directory if it doesn't exist: hdfs://ha/warehouse/tablespace/managed/hive/test_database.db/test_table/.hive-staging_hive_2024-10-16_15-21-40_061_8849017887411502804-324/10/16 15:21:40 INFO FileOutputCommitter: File Output Committer Algorithm version is 124/10/16 15:21:40 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:f