Q&A

Hi, My issue related to committing workflow changes onto my Github repository. I have made changes in the providers.xml file and now my Nifi flows are linked to Github flowPersistenceProvider.However, when I try to commit changes to the same linked repo, it gives me below error:  and in the log files, it says: "Flow version is not added yet, can not be serialized.. Returning Conflict response."I also witnessed on "show local changes" that the prompt gives an error "version must be greater than 1". Are these linked?Any guideline would help. Thanks. 

Dear FreindsMy Cloudera Cluster has no access to the internet - if I need to gather some parcel or any file via wget for example But what if I want to import some python library in HUE or Zeppelin that requires dynamic access to the internet.I have installed via pop3 pgeocode library and now in Peppelin or HUE when I want to dynamically call that library I am getting TIme OUT issue  : File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_openraise URLError(err)urllib.error.URLError: <urlopen error [Errno 110] Connection timed outHow can I use proxy server in such cases so HUE or Zeppelin if record has access to the internet - able to reach the python library modules.

Hello I am using the EnforceOrder in order to sort the flow files. This procesosr is maintaning state and i want to clear in it some regular interval for e.g. 4 min. I am runnig NIFI over a cluster. I am clearing the state through Inactive Timeout propery by setting it 4 min. I can see some time state is clear and most of the time not. Please help in a resolution for how clear state consistently.

I am using NiFi 2.0 that supports python scripts directly from a working directory, after I add my custom processor to the canvas, an error "Processor invalid because initializing runtime environment for the processor" is indicated in each custom processor. I have already installed Java JDK 21, python 3.11 and all the requirements specified in the official documentation. How can I resolve this?I have already installed Java JDK 21, python 3.11 and all the requirements specified in the official documentation.I changed the python extension parameters in the nifi.properties file.This is what appears in the app-log:2024-07-24 10:31:06,608 ERROR [Initialize CreateFlowFile] org.apache.nifi.NiFi An Unknown Error Occurred in Thread VirtualThread[#138,Initialize CreateFlowFile]/runnable@ForkJoinPool-1-worker-2java.lang.RuntimeException: Failed to launch Process for Python Processor [CreateFlowFile] Version [0.0.1-SNAPSHOT]at org.apache.nifi.py4j.StandardPythonBridge.getProcessForNextComponent(St

Unfortunately, Cloudera seems to only actively maintain a maven plugin for packaging the code into a nar file. My company has a standard to do things in gradle and do not want me changing to a maven project. Previously, we used a project compatible with gradle 6. Now we are upgrading to Java 17 and Gradle 8.8 and what we had previously used is now incompatible so I was wondering if anyone has had luck with this issue?

I want to grant user 'txw387' RWC permissions. There is a hadoop folder 'txw387' under /user.However, this error occurs: ERROR: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 403 Error: Grantee user txw387 doesn't existat org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preGrant(RangerAuthorizationCoprocessor.java:1277)at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preGrant(RangerAuthorizationCoprocessor.java:1110)at org.apache.hadoop.hbase.master.MasterCoprocessorHost$164.call(MasterCoprocessorHost.java:1892)at org.apache.hadoop.hbase.master.MasterCoprocessorHost$164.call(MasterCoprocessorHost.java:1889)at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preGrant(MasterCoprocessorHost.java:1889)at org.

Dear Cloudera Support Team,I am experiencing an issue when running the ImportTsv command in HBase using the following command:hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator='\t' -Dimporttsv.columns='HBASE_ROW_KEY,cf:gender,cf:age,cf:income,cf:exp' Customers /hbase/data.txtThe job is failing with the following error message:2024-10-03T15:00:41,530 INFO [main] mapreduce.Job: Running job: job_1727859616916_00042024-10-03T15:00:50,594 INFO [main] mapreduce.Job: Job job_1727859616916_0004 running in uber mode : false2024-10-03T15:00:50,597 INFO [main] mapreduce.Job: map 0% reduce 0%2024-10-03T15:00:50,616 INFO [main] mapreduce.Job: 9616916_0004_02_000001Exit code: 1[2024-10-03 15:00:49.597]Container exited with a non-zero exit code 1. Error file: prelaunch.err.Last 4096 bytes of prelaunch.err :Last 4096 bytes of stderr :SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.s

Hi, I am looking for explanation for below error. Why driver is trying to convert LIMIT into TOP? This log comes from LogLevel=5.Sep 18 16:51:32.339 DEBUG 73090 com.cloudera.impala.jdbc.core..call: establishConnection() in a new thread: 73090Sep 18 16:51:32.342 INFO 73090 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: Schema is set to default: defaultSep 18 16:51:32.343 DEBUG 73090 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: socketTimeout = 0, loginTimeout = 60Sep 18 16:51:32.343 DEBUG 73090 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: SocketTimeout is: 60 seconds for testSep 18 16:51:34.408 DEBUG 73090 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: socketTimeout is set back to 0 secondsSep 18 16:51:34.408 DEBUG 73090 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.setServersideProperties: {LoginTimeout=60, SSLTrustStoreType=JKS}

Hi, I am looking for explanation for below error. It fails on getColumns call.  Caused by: java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500001) Error setting metadata information: Metadata Initialization Error. at com.cloudera.impala.hivecommon.dataengine.metadata.HiveJDBCColumnsMetadataSource.initializeColumn(Unknown Source) at com.cloudera.impala.hivecommon.dataengine.metadata.HiveJDBCColumnsMetadataSource.(Unknown Source) at com.cloudera.impala.hivecommon.dataengine.HiveJDBCDataEngine.makeNewMetadataSource(Unknown Source) at com.cloudera.impala.dsi.dataengine.impl.DSIDataEngine.makeNewMetadataResult(Unknown Source) at com.cloudera.impala.hivecommon.dataengine.HiveJDBCDataEngine.makeNewMetadataResult(Unknown Source) at com.cloudera.impala.jdbc.jdbc41.S41DatabaseMetaData.createMetaDataResult(Unknown Source) at com.cloudera.impala.jdbc.common.SDatabaseMetaData.getColumns(Unknown Source)...Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Cloudera][Im

Spark on HDP-3.1 is copying hive-1.x jars in /usr/hdp/3.1.4.0-315/spark2/jars/ directory. Due to this I am getting classpath issues when I tried to interact with hive-3 server via my spark code. Any specific reason why spark2 is still using hive-1.x jars. I also am supplying hive3.x jars via --jars option but no effect. Since hive-1.x jars are added first to the classpath all the classes are loaded from hive-1.x jar by AppClassLoader. After moving them into /tmp/ directory all the jobs are working fine because MutableUrlClassLoader is loading classes from hive-3.x jars. Deleting them from /usr/hdp/3.1.4.0-315/spark2/jars/  is not a good solution. Is there any workaround to solve this issue. Any help would be appreciated. Thanks!

Hello . I get the error message saying : nifi-registry 2024-10-02 08:08:58,582 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-10-02 08:08:58,581 INFO [NiFi Registry Web Server-37] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[cef567], groups[] does not have permission to access the requested resource. Unable to view users/user groups. Ret ││ urning Forbidden response. ││ nifi-registry 2024-10-02 08:08:58,651 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-10-02 08:08:58,650 INFO [NiFi Registry Web Server-52] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[cef567], groups[] does not have permission to access the requested resource. Unable to view use

Hello everyone,We would like to try disabling Hive Kerberos on CDP 7.1.7.Here are the parameters we tried disabling:Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xmlhive.server2.authentication.kerberos.principal hive.server2.authentication.kerberos.keytabhive.server2.authentication NONEhive.security.authorization.enabled falsehive.server2.enable.impersonation falsehive.metastore.sasl.enabled falseHive Service Advanced Configuration Snippet (Safety Valve) for core-site.xmlhadoop.security.authentication simplehadoop.security.authorization falseAfter turning off the following parameters and restarting HiveServer2, we noticed that HiveServer2 still seems to fail due to Kerberos authentication.Could you please advise on how to completely disable Kerberos for HiveServer2? Thank you.  

Hi Experts,Please let us know if we can use any NiFi processors for performing MSCK repair on the hive table to reflect the data that are updated by NiFi to HDFS. Thanks!Command : MSCK REPAIR TABLE tblName;

Hi guys, I'm relatively new to Apache Nifi, and there's a subject that I don't understand the logic behind, how to handle flowfiles between processors.Basically I need to extract the data from an ExecuteSQL processor, then execute another ExecuteSQL to delete the data from the destination and finally insert the data using PutDatabaseRecord, but the value of the first ExecuteSQL is overlapped by the second, and it needs to be done in that order to ensure that the data exists before deleting the old base, could anyone guide me?

Dear Cloudera Community,I am currently testing HBase for bulk loading purposes, and for this, we use the pseudo-distributed mode (single-host standalone) as our configuration. After completing all the configurations, I executed the jps command, which displayed all the necessary processes for testing. [super@dc1-apache-hbase bin]$ jps1119857 Jps1118448 NodeManager1119235 HQuorumPeer1117730 NameNode1117863 DataNode1119627 HRegionServer1118125 SecondaryNameNode1119405 HMaster1118332 ResourceManagerAdditionally, I did not use Zookeeper manually for the process; instead, I used export HBASE_MANAGES_ZK=true.However, once I enter the HBase shell, the HMaster process disappears, and I encounter the following error message:"ERROR: KeeperErrorCode = NoNode for /hbase/master"[super@dc1-apache-hbase conf]$ jps1122225 Jps1118448 NodeManager1121283 HQuorumPeer1122019 JarBootstrapMain1117730 NameNode1117863 DataNode1121675 HRegionServer1118125 SecondaryNameNode1118332 ResourceManagerI have attac

I need to use OAuth 2.0 to publish messages to a Confluent Kafka topic. NiFi doesn't have support for this out of the box, so before I build my own solution, I was wondering if anyone knew of any solutions currently being used for this that they'd be willing to share, or any existing workaround.I'm using NiFi version 2.0.0-M4. Thanks

Hello, I have been getting the below error when i am trying to startup my nifii registry : Caused by: org.springframework.ldap.UncategorizedLdapException: Uncategorized exception occured during LDAP processing; nested exception is javax.naming.NamingException: [LDAP: error code 1 - 00000000: LdapErr: DSID-0C091543, comment: TLS or SSL already in effect, data 0, v4f7c]; remaining name '' at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:228) at org.springframework.ldap.core.support.AbstractContextSource.doGetContext(AbstractContextSource.java:155) at org.springframework.ldap.core.support.AbstractContextSource.getReadOnlyContext(AbstractContextSource.java:166) at org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider.load(LdapUserGroupProvider.java:447) at org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider.onConfigured(LdapUserGroupProvider.java:377) at org.apache.nifi.regi

I want to establish an ODBC connection to my Impala data warehouse in a CML project. I'm running a CML session and have configured my odbc.ini and odbcinst.ini files, but I need to install the driver. Everything I see here https://www.cloudera.com/downloads/connectors/impala/odbc/2-7-2.html describes installing it via the installation wizard.Since CML runs a linux pod, I tried downloading the .deb file for the ODBC driving and then executing `dpkg clouderaimpalaodbc_2.7.2.1011-2_amd64.deb` but it has to be as root. If I run `su` it asks for a password. I'm not sure how this password gets generated by the system. I tried my workload password and portal password but neither worked.Is there a more principled way of connecting via ODBC in CML or is there a way to run as root in CML?

In HBase, i have a column qualifier in which i have a data like below:ReportV10\x00\x00\x00\x00\x02\x02\x02When i am reading this table from spark using shc connect, i am getting junk characters in result. Below is the piece of code i am using to read a HBase table:catalog='''{"table":{"namespace":"db1","name":"tb1"},"rowkey":"key","columns":{"rowkey":{"cf":"rowkey","col":"key","type":"string"},"nf_hh0":{"cf":"nf","col":"hh0","type":"string"}}}'''df=spark.read.option("catalog",catalog).format("org.apache.spark.sql.execution.datasources.hbase").load()df.show(1,False)+------------------------------------------------+| rowkey                          | nf_hh0                | +---------------------------+------------------- +|26273707950926220...|ReportV10��   |+---------------------------+--------------------+Spark version: 2.3.2.3.1.0.319-3HBase version: 2.0.

Hi everybody!I am struggling trying to create a NiFi Kubernetes cluster out of cetic helm chart. After installation, I see a working zookeeper cluster (3 instances) and  a 2 instances stateful set for NiFi. One instance is correctly elected as cluster coordinator, NiFi frontend shows me a 1 member cluster. The second instance's log keeps on popping messages such as:2022-07-01 11:52:23,839 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message2022-07-01 11:52:29,313 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message2022-07-01 11:52:34,792 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.P

hadoop distcp -i -log /tmp/ hdfs://xxx:8020/apps/yyyy hdfs://xxx_cid/tmp/ In this case the "xxx" is the "un-secure" cluster, while "xxx_cid" in the secure cluster. We are launching the job from the Kerberos cluster, with the appropriate kinit for the user and getting the following error: java.io.IOException: Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.; Host Details : local host is: "xxx/10.x.x.x"; destination host is: "xxx":8020; ... Caused by: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections. I thought by launching the job from the secure cluster, that we could avoid any access issues. But it appears that the processes are kicked off from the "source" cluster. In this case, that's the insecure cluster. Idea's on getting around this?

Dear Cloudera Community,We are currently conducting warehouse testing using Apache HBase and need to load large files into HBase tables. Could you kindly suggest any tools or specifically designed for bulk loading large datasets into HBase?Thank You!

Hello,I am currently testing HBase for bulk loading purposes, and I am using the pseudo-distributed mode (single-host standalone) for my configuration. After completing all the configurations, I ran the jps command, which displayed all the necessary processes for testing, as shown below:[super@dc1-apache-hbase bin]$ jps25808 ResourceManager25585 SecondaryNameNode36961 HMaster25203 NameNode37347 Jps25925 NodeManager36805 HQuorumPeer25334 DataNode37135 HRegionServer[super@dc1-apache-hbase bin]$I did not use Zookeeper manually but instead set export HBASE_MANAGES_ZK=true. However, once I enter the HBase shell, I can list the tables but cannot create any new tables. The error displayed is as follows:hbase:001:0> listTABLE0 row(s)Took 0.7732 seconds=> []hbase:002:0>hbase:003:0> create 'table_1','cf'ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3215)at org.apache.hadoop.hbase.maste

Hi, experts,I need to insert data from CSV to DB. CSV file name is the same as DB table name, but table has additional primery key column, so I need to add such column in CSV. I can get max key value from DB table. Please, advise how to use this key value in flowfile. Here is my flowThanks in advance.

Hello , I am trying to setup LDAP on my Nifi Registry and I am getting the below errror : nifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Caused by: java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry/nifi-registry-current/conf/login-identity-providers.xmlnifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut at org.apache.nifi.registry.security.authentication.IdentityProviderFactory.loadLoginIdentityProvidersConfiguration(IdentityProviderFactory.java:160)nifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut at org.apache.nifi.registry.security.authentication.IdentityProviderFactory.getIdentityProvider(IdentityProviderFactory.java:110)nifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut at org.apache.nifi.registry.security.authentic