Solved

Hello,I am currently testing HBase for bulk loading purposes, and I am using the pseudo-distributed mode (single-host standalone) for my configuration. After completing all the configurations, I ran the jps command, which displayed all the necessary processes for testing, as shown below:[super@dc1-apache-hbase bin]$ jps25808 ResourceManager25585 SecondaryNameNode36961 HMaster25203 NameNode37347 Jps25925 NodeManager36805 HQuorumPeer25334 DataNode37135 HRegionServer[super@dc1-apache-hbase bin]$I did not use Zookeeper manually but instead set export HBASE_MANAGES_ZK=true. However, once I enter the HBase shell, I can list the tables but cannot create any new tables. The error displayed is as follows:hbase:001:0> listTABLE0 row(s)Took 0.7732 seconds=> []hbase:002:0>hbase:003:0> create 'table_1','cf'ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3215)at org.apache.hadoop.hbase.maste

Hi, experts,I need to insert data from CSV to DB. CSV file name is the same as DB table name, but table has additional primery key column, so I need to add such column in CSV. I can get max key value from DB table. Please, advise how to use this key value in flowfile. Here is my flowThanks in advance.

Hello , I am trying to setup LDAP on my Nifi Registry and I am getting the below errror : nifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Caused by: java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry/nifi-registry-current/conf/login-identity-providers.xmlnifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut at org.apache.nifi.registry.security.authentication.IdentityProviderFactory.loadLoginIdentityProvidersConfiguration(IdentityProviderFactory.java:160)nifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut at org.apache.nifi.registry.security.authentication.IdentityProviderFactory.getIdentityProvider(IdentityProviderFactory.java:110)nifi-registry 2024-09-27 09:25:06,919 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut at org.apache.nifi.registry.security.authentic

Hello,I am unable to access Hive using Beeline connection string in the edge node. The JDBC String that I am using is as follows:beeline -u "jdbc:hive2://Example.com:10000/default;principal=hive/Example.com@EXAMPLE.COM" Error:Unknown HS2 problem when communicating with Thrift server.Error:jdbc:hive2://Example.com:10000/default;principal=hive/Example.com@EXAMPLE.COM: Invalid status 21 (state=08S01,code=0)Help is required to fix this issue. Thank you. 

Hi,I am using NiFi 1.26.0 and having a flow with combination of ListFile and FetchFile.Basically i have three files in my local directory (test_1.txt, test_2.txt,test_3.txt). I have to fetch only one file (ex:test_1.txt) for further process.Based on regex listFile list's all three files.But facing issue in fetch file. Instead of fetching only one file, it tried to fetch all three files listed by listFile processor  and giving output as three files instead of one file, but content of all three files are same as content of file mentioned in fetchFile configuration (test_1.txt)In data provenance could see it is trying to fetch the files which is not mentioned in fetchFile configuration and after fetching the files it modifies the content.How can i fix this. Please help.

I recently downloaded the latest version of NiFi (nifi-2.0.0-M4) and installed it by placing the folder in /opt/nifi-2.0.0-M4. The only modifications I made were in the nifi-env.sh and nifi.properties files as follows:nifi-env.shexport JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64nifi.properties# Site to Site propertiesnifi.remote.input.host=10.37.10.199nifi.remote.input.secure=falsenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=60 secnifi.remote.contents.cache.expiration=60 secs# web properties ############################################### For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.# Be aware that changing these properties may affect how your instance can be accessed without any restriction.# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.nifi.web.http.host= 0.0.0.0 # Here goes the IP of my virtual mach

Hello community,I want to get the from hdfs to confluent cloud kafka is there any suggestion how to do that.

I have created a three node NIFI cluster and the embedded zookeeper is set up and all the three nodes are communicating with each other through heartbeats but when I try to access the NIFI UI I am getting the below error:Unauthorized error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Signed JWT rejected: Another algorithm expected, or no matching key(s) found", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"How do I fix this issue?

Hi,regarding the disks of an HDFS datanode with for example 10 disks, what are good values ​​for the read/write throughput for each JBOD disk? Are there any benchmarks for a production environment? (e.g. 200 MB/sec).Thank you.

Hi all,I'm trying to get our CDSW 1.10.5 Instance to recognize the GPUs, according to this documentation: https://docs.cloudera.com/cdsw/1.10.5/gpu/topics/cdsw-gpu.htmlBasically: nvidia-smi shows the GPUs are present on the host but after activating GPU-Support in Cloudera Manager the GPUs do not show up in the CDSW interface. We tried to follow this advice, since we also run an AirGapped Setup: https://community.cloudera.com/t5/Support-Questions/CDSW-1-6-does-not-recognize-NVIDA-GPUs/td-p/280207but unfortunatelly retagging the image did not solve the issue. How can we obtain additional info to diagnose the issue?

Hi allI was looking for some help regarding moving a NiFi Registry set up from one server to another.At the moment we set up a kind of "QA" system which was running well and so we have now created the production server if you like which we're going to use for all of our NiFi connections. The current registry is backed by a Git Repo using BitBucket which all works as expected.We have set up the new server, installed registry and connected the same repo and all seems to be fine in terms of the NiFi servers can connect and I can see registry on the new server.What I want to do now though, is move everything we had in the "QA" registry server, into the production server if this is possible (so that means the buckets, versions and flows etc) but JUST for the registry (we will then pull these fresh into the NiFi server(s)).I think I read somewhere that the information in registry is kinda "linked" to a registry server via a UUID or something but I'm not sure how that works (if not, my assump

Hi all, I am practicing spark. When using pyspark to query table in Hive, I can retrieve the data from an external table but query a internal table. Here is the error message:>>> spark.read.table("exams").count()23/03/30 22:28:50 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not existHive Session ID = eb0a9583-da34-4c85-9a1b-db790d126fb1Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/spark/python/pyspark/sql/readwriter.py", line 301, in table return self._df(self._jreader.table(tableName)) File "/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", File "/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/spark/python/pyspark/sql/utils.py", line 69, in deco raise AnalysisException(s.split(': ', 1)[1], stackTrace)pyspark.sql.utils.AnalysisException: u'\nSpark has no access to

Update Attribute processor Not working after upgrade from version 1.8 to version 1.9. We recently upgraded our Nifi Cluster from version 1.8 to version 1.9 and we were surprised to see the update attribute processor all going invalid for our existing flows. We had to manually fix the existing flow but this ideally should have been handled with the upgrade .Is there a way to overcome this ?

Hello everybody, I have a ODBC (DSN=IMPALA_PRD) setup to connect to Impala server through kerberos authentication.I have created the ticket in MIT with success.I've tested the connection with success.However when using the same ODBC connection on a python script or a PowerBI dashboard, it returns the error: ODBC: ERROR [HY000] [Cloudera][DriverSupport] (1110) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct and confirm if SSL should be enabled for the connection.\r\nERROR [HY000] [Cloudera][DriverSupport] (1110) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct and confirm if SSL should be enabled for the connection. For quick test i'm using this simple script:import pyodbcconn_impala = pyodbc.connect(DSN="IMPALA_PRD", autocommit = True)cursor = conn_impala.cursor()Could you please help me solve this riddle?Best regards, 

Some Scripts suddenly stopping without ending for some days

We are working on converting the Unix time into date-time using the QueryRecord processor. We understand that QueryRecord is built with the calcite SQL syntax and tried to use the DATE_FROM_UNIX_DATE function. We are receiving the below error. Please help us resolve this issue with your expertise.We can achieve this conversion using the update attribute but we wanted this as part of the Query record as we wanted to apply this conversion on the completed dataset.Query :SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFileError :QueryRecord[id=2cc603e4-0191-1000-f3c7-ef830564cf72] Unable to query FlowFile[filename=convertfunctionsample.csv] due to org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile]- Caused by: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile]- Caused by: org.apache.calcite.runtime.Ca

I am reading oracle table using pySpark. my oracle table contains data type NUMBER and it contains 35 digits long value. when I read this column using spark, it seems spark assumes more precision than original (+10 precision)and end up in throwing following error:java.lang.ArithmeticException: Decimal precision 45 exceeds max precision 38`please note: I am inferring schema from spark data frame as I don't want to explicitly pass schema.code:df_rdbms_data = spark.read.jdbc(url=source_jdbc_url, table=source_table_name, properties=source_properties)df_rdbms_data.printSchema()df_rdbms_data.show()Is there any way to resolve this issue apart from not passing explicit schema or explicit casting?Thanks in advance.

Hi all,After establishing a connection to Cloudera Virtual Warehouse from PowerBI Desktop using Hive LLAP connector, I'm not able to load Iceberg tables. I receive the following error:DataSource.Error : ODBC : ERROR [42000] [Microsoft][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: Error while compiling statement: FAILED: Runtime Exception org.apache.hadoop.hive.ql.metadata.HiveException: Table XXXX is an Iceberg table format which is not currently supported.I tried also installing the Hive ODBC driver 2.7.0. I configured the User DSN and test was successful. When I try to connect form PBI Desktop, I can establish the connection but I receive the same error if I try to load an Iceberg table:DataSource.Error: ODBC: ERROR [42000] [Cloudera][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: Error while compiling statement: FAILED: Runtime Exception org.apache.ha

hi cloudera community,creating a full template using the nifi webui mode works perfectlywe need to create the template, export and import using the nifi api, has anyone managed to configure this?

Our group has tried connect janusgraph with Cloudera HBase, but it fails.  In the documentation of CDP 7.2.18, these two Spark features: GraphX and GraphFrames are not supported by Cloudera. Is there any other Graph Data Base or Graph computation tool is supported by Cloudera?

Greetings, I am trying to migrate from Nifi 1.16.2 to 1.16.3.When I try to use my flow file form my 1.16.2 installation, the application goes up but the controller service StandardRestrictedSSLContextService 1.18.0, shows a problem with the sensitive properties that are not supported.After updating anything inside the controller, everything starts working as expected. But if a restarts Nifi, the application does not restarts.I try to find some errors in the  nifi-app.log but none is logged, just Warnings, here some of them  2022-11-23 17:01:00,812 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 9, col 21: cvc-complex-type.2.4.a: Invalid content was found starting with element 'description'. One of '{url}' is expected.2022-11-23 17:01:00,825 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 71, col 19: cvc-complex-type.2.4.d: Invalid content was foun

Using NiFi, I am processing a file. Once processing is done, I want to shift it to an archive folder. I want it to keep only say 30 days files and older files to be deleted. How to do. One more question is how to have timestamp name appended to file name. Like if file name is file1.txt then how to have it renamed like file1_09_09_2024_15_54.txt where we have 09_09_2024 as date and 15_54 as time(hour and mins)

HiI have a question on our Nifi flow design and specifically about the usage of DuplicateFlowFile processor.The latest documentation mentions that, it is intended for load test.Does that mean, it can not be used in regular production flows? Please confirm.https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/nifi-2.0.0-M4/org.apache.nifi.processors.standard.DuplicateFlowFile/index.html"Intended for load testing, this processor will create the configured number of copies of each incoming FlowFile." Please verify the current and improved designs below and advise if any implications with the new approach (where DuplicateFlowFile processor is used to clone flow files)Current Design:  New Design is below (with DuplicateFlowFile processor)        

Say I have a file having 100 records. In this file I want to ignore the first line as it's data I do not want to process (It contains some info like file name, date and number of records). I want to process from second line. How to do that?

I am facing an issue with my Nifi as I am unable to access the Nifi anymore. I think this maybe because I have modified and replaced the nifi.properties file from the original downloaded version of the file.When I try to run Nifi as a service using sudo service nifi start it does not start the service.Here are the nifi-app.log that I can see...   ERROR [main] org.apache.nifi.NiFi Failure to launch NiFijava.lang.IllegalArgumentException: There was an issue loading properties at org.apache.nifi.NiFi.initializeProperties(NiFi.java:361) at org.apache.nifi.NiFi.convertArgumentsToValidatedNiFiProperties(NiFi.java:343) at org.apache.nifi.NiFi.convertArgumentsToValidatedNiFiProperties(NiFi.java:339) at org.apache.nifi.NiFi.main(NiFi.java:331)Caused by: java.lang.IllegalStateException: Sensitive Properties Key [nifi.sensitive.props.key] not found: See Admin Guide section [Updating t at org.apache.nifi.properties.NiFiPropertiesLoader.getDefaultPr