Q&A

Hi! I am trying to connect to a MySQL RDS instance in AWS with Apache NiFi on Linux Ubuntu. The MySQL database is behind a SSH tunnel which is working in a dedicated terminal. The MySQL database is accessible through another Linux terminal. $ ssh -i /home/bernardo/.ssh/sshbernardo3.pem -N -L 33061:<db_name>.<id>.eu-west-1.rds.amazonaws.com:3306 <user>@<ip_address> -p <port> I have uploaded and unzipped the JDBC driver from https://downloads.mysql.com/archives/c-j/ The file java.sql.Driver is located in the directory /usr/share/java/META-INF/services in Linux I have installed the jdbc connector in Linux and set up the user to use JDBC$ sudo apt-get install libmysql-java$ cd /home/bernardo$ gedit .bashrc Added the following lines at the end of the file: # Setting up the user to use JDBCCLASSPATH=$CLASSPATH:/usr/share/java/mysql-connector-java-8.0.26.jarexport CLASSPATH $ source .bashrc In NiFi I have used a D

Hello community,  I would like to configure a StandardSSLContextService for kafka to read the password for the keystore from an environment variable. Sadly it is not possible to use the "Expression Language" for this sensitive property. Background: We are using an operator to provision kafka users. The operator generate a kubernetes secret with the keystore and keystore password. The kubernetes secret is mounted into the nifi container and can be referenced in the StandardSSLContextService. Currently we have to add the password for this keystore manually. But when the kafka user certificate expires and is rotated, the password of the keystore changes too. I am looking for a solution to read the password automatically, e.g. by setting it as environment variable in nifi. Any idea how I could achive this? Is there a mechanism I am missing?Thank you a lot for your help,Constantin 

Dears,Currently we are using Apache kafka 2.13-2.6.0 Version in our production and Currently (log4j-1.2.17) is installed on the server.After a recent security scan, our vendor suggested upgrading to Log4j version 2.16.0 or higher since 1.x is an Unsupported Version(end of life) and a CVE-2021-4104 vulnerability.Could you please suggest and provide the guidance to upgrade the log4j version at the earliest.

I am getting a 502 Bad Gateway error when I try to load the Apache Ambari UI at localhost:8080.

My Nifi instance of 5 node is not starting with the below error , please help on resolving this , I have tried deleting the repos and setting nifi.flowcontroller.autoResumeState=false in nifi.properties , start nifi service then also getting the same error , please help on this I have also increased the heap size that also did not resolve this 2021-11-24 14:20:43,413 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError: nullat java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)at org.apache.nifi.util.file.FileUtils.copy(FileUtils.java:304)at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:110)at org.apache.nifi.controll

Scenario:I have a List File Processor looking out for incoming file ("file 1"). I scheduled it to start picking up the "file 1" 20s after the "file 1" is downloaded.Let assume that the List File Processor noticed the incoming "file 1" and started the delay 20 sec before picking the "file 1".In the middle of 20 sec delay, there is an new incoming file ("file 2") noticed by the List File Processor.List File Processor will be interrupt and reset the 20 sec schedule time when ("file 2") appear within the previous 20 sec delayCould the List File Processor in the middle of 20 sec delay be interrupt and restart the initial delay? Appreciate help with explanation and example.Thanks.

My environment contains two separate and independent setups. My development environment runs all software on a VM with everything being local. Either using localhost or my internal IP address. My production environment runs with everything being in a pod. Each having a hostname. Currently when I create a template from my development environment and move it to my production, I have to go through all the processes and change the endpoints so that it will continue to run and process. Is there a method in NiFi, that I can define the endpoints on the system that contains the NiFi service, that the processes can rely on to get the proper value for the endpoints. I can move the templates between my development and production and visa-versa without having to reconfigure each process? Thank you.

Guys, I just wondering does the Data Labeling and Data Classification works whenever the data travels around? My main goals is to protect the data is being misused. For example I can block any data that already labelled and classified as Confidential being printed or copied to USB drives. Appreciate your kind insight for this.

Hi, I noticed that in NiFi rest api documentation for version 1.9.0 the parameter context endpoints have been removed, even though they existed in version 1.15.2Furthermore, I am using NiFi version 1.13.2 and I can see that in my browser's debugging tools, in the network tab, whenever I create a parameter context, a post request is sent to http://localhost:8080/nifi-api/parameter-contexts: and the following payload is being sent :{ "component": { "name": "aaaaaaaaaaa", "description": "aaaaaaaaaaaaaaaa", "parameters": [] }, "revision": { "clientId": "3d3697c2-017e-1000-0610-31b28073cf9b", "version": 0 }} The parameter context is successfully created. However when I attempt to send a post request from an external API or even hit the previously mentioned endpoint through the browser using the same payload, I get:HTTP 405 Method Not Allowed My questions are:1. Is anyone aware whether these endpoints have been removed/r

Hi, I want to create a hive table which will store data with orc format and snappy compression. Will power bi be able to read from that table? Also do you suggest any other format/compression for my table?

I am using nifi in production environment and I need to run it in a 3-node cluster on docker (3 separate VMs). Have used the following environment variables for Nifi:   version: '3.2'  services:    zookeeper:    hostname: zookeeper    image: bitnami/zookeeper:latest    container_name: collections_zookeeper    ports:      - "22181:2181"    environment:      - ALLOW_ANONYMOUS_LOGIN=yes   nifi:    build:      context: ./nifi      dockerfile: Dockerfile    image: apache/nifi:custom    ports:      - 8080    environment:      - NIFI_WEB_HTTP_PORT=8080      - NIFI_CLUSTER_IS_NODE=true      - NIFI_CLUSTER_NODE_ADDRESS=1      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082      - NIFI_ZK_CONNECT_STRING=zookeeper:22181      - NIF

I have an ETL flow which transfers data from a hive table to another through pyspark. The tables are partitioned. Although I see that in the partition's path in HDFS there are small parquet files. I want to ask:1)How can I merge these files?2)Is there any max size or recommended size for hive partitions?

Hello everybody, Recently my DBA team called me informing that Sqoop was sending queries to the database without parameter binding. Looking in the Sqoop documentation I got confirmation of the use of Prepared Statement by Sqoop only in the export command, with --batch parameter. With import command there is not this option to use. Could you help me identify how I could (if possible) use Sqoop's parameters to make it use binding variables in import querys? Any thoughts will be appreciated.Thanks! PS: Above is the description of the --batch parameter.--batch Use batch mode for underlying statement execution.

Recently we have deploy a new hdp cluster into 3 master and 5 worker nodes.But it looks like there are several component that mis-placed such as Hbase service.A Hbase master has been installed into a worker node and 3 Hbase Region Servers is located on 3 master nodes.So, we've a plan to relocate the some mis-placed component of hbase (the hbase master must be in master node and the region servers mus be in worker nodes).Our first step is to  decomissioned the 3 hbase region server on masternode but it cannot be done because the hbase master failed to be startedThen we investigate the hbase master log and we found 2021-11-30 13:42:24,379 ERROR [master/DCHDPD03:16000:becomeActiveMaster] master.HMaster: Failed to become active masterorg.apache.hadoop.hbase.util.FileSystemVersionException: hbase.version file is missing. Is your hbase.rootdir valid? You can restore hbase.version file by running 'HBCK2 filesystem -fix'. See https://github.com/apache/hbase-operator-tools/tree/master

Accessing the URL on port 18088 returns a ERR_SSL_PROTOCOL_ERROR error in Chrome. history server logs have the following: 11:25:29.130 AM WARN HttpParser Illegal character 0x16 in state=START for buffer HeapByteBuffer@1af6bd16[p=1,l=517,c=8192,r=516]={\x16<<< ...}11:25:29.130 AM WARN HttpParser Illegal character 0x16 in state=START for buffer HeapByteBuffer@21158554[p=1,l=517,c=8192,r=516]={\x16<<< ...}11:25:29.130 AM WARN HttpParser bad HTTP parsed: 400 Illegal character 0x16 for HttpChannelOverHttp@2390a367{r=0,c=false,a=IDLE,uri=null}11:25:29.131 AM WARN HttpParser bad HTTP parsed: 400 Illegal character 0x16 for HttpChannelOverHttp@7843db5f{r=0,c=false,a=IDLE,uri=null} Note: some lines truncated to avoid exposing data. There was a similar post regarding Schema Registry UI in Ambari but the solution doesn't appear to be applicable here (though I suspect the issue is similar). Any thoughts?

I use version CDP Private Cloud Base7.1.4(Private Cloud) now, but I don't find any information about high availability in the document; But in Enterprise 6.3.4 (https://docs.cloudera.com/documentation/enterprise/latest/topics/admin_cm_ha_overview.html) and Cloudera Manager7.5.1(Private Cloud • PCE 1.3.1) Version (https://docs.cloudera.com/cloudera-manager/7.5.1/managing-clusters/topics/cm-high-availability.html) can see that there is highly available information. Which scheme should I follow?

When CDH cluster adds sentry service, an error appears in the above picture, indicating that HDFS or core configuration is required (but this service also needs to be configured with core.defaultFs). I want to know whether HDFS is generally necessary for sentry service, but I don't see relevant  instructions (Prerequisites) on the official website (Before You Install Sentry | 6.3.x | Cloudera Documentation). Can anyone help provide relevant documents?  because I need to convince customers that HDFS is necessary.  Thankssss!!!

We have a lab cluster. We moved the MySQL DB to Oracle Cloud. The on-prem DBs remained but are now several months old. I am trying to revert back to the on-prem without copying the data back from OCI. The loss of changes in the several months does not matter as this is a lab cluster. The issue is the license was updated in between these DB changes and now SCM will not start. FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Upgrade not allowed in Express or Trial mode What table has the license info. I can grab just that table from OCI?

I am new with Cloudera and we are currently running CDP Datalake and CDP Datahub 7.2.6. I am trying to follow the instruction to do the mitigation for log4j (TSB 2021-545) but I could not follow the instruction to navigate to Cloudera Manager. I can go as far as the Cloudera management Console but could not see the Cloudera Manager. Can you please help me or send me the URL. 

I need to convert 2021-10-03 15:10:00.0 as 2021-10-03T15:10:00-04:00I tried with. from_utc_timestamp(from_unixtime(unix_timestamp('2021-10-03 15:10:00.0', "yyyy-MM-dd HH:mm:ss.S"),"yyyy-MM-dd'T'HH:mm:ssXXX"),"America/New_York") I got Null valueAny suggestions please

Hi to all,My cluster is the latest CM 7.4.4, and CR 7.1.7.Cluster working fine until I enable Kerberos. Zookeeper wont start with error Could not configure server because SASL configuration did not allow the Zookeeper server to authenticat itself properly:javax.security.auth.login.LoginException: Message stream modified (41) I'm able to get zookeeper and other services up if I commented# renew_lifetime = 7d  on all the nodes and kerberos server. But only Hue Kerberos Ticket Renewer will have a problem. So what I did was I commented out renew_lifetime = 7d on server that hosted Kerberos Ticket REnewar roles. So now my cluster will be up. But this does not like a good workarund as some of the UI are having a problem like Atlas and Solr with error (tgt renewal). Anyone encounter this? P/S: I have a working Kerberized cluster with same version of CDP. It is working fine. Same exact version, os version, java version, and kerberos version. Only not all comp

Hi while running '/opt/cloudera/cm/schema/scm_prepare_database.sh  mysql scm scm', I'm getting below error. I'm unable to find the cause. Please help me resolve this error.  JAVA_HOME=/usr/java/latestVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/java/latest/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Thu Dec 30 05:34:34 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors t>Dec 22 04:06:46 vhcccloudera01d cm-server[771896]: 04:06:46.929 [main] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: relation "cm_version"> I am trying to run cloudera manager but  my ip:7180  shows "cant be reached "can anyone please tell how to resolve this

Hello All,I am trying to install the CDP Private Cloud trial version of the Cloudera agent on RHEL 8.5 and was getting a few dependency errors. After installing all the requirements, initial errors were gone but somehow below error is not getting resolved even though we have installed the IPREF3 ( iperf3-3.5-6.el8.i686.rpm) package but still getting the error for "libiperf.so.0". Below is the command we are running:sudo yum --disablerepo=* --enablerepo=cloudera*  install cloudera-manager-agentUpdating Subscription Management repositories.Last metadata expiration check: 0:26:55 ago on Thu 23 Dec 2021 11:47:10 AM EST.Error: Problem: conflicting requests  - nothing provides libiperf.so.0()(64bit) needed by cloudera-manager-agent-7.4.4-15850731.el8.x86_64(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) I searched the forum but did not get an answer hence posting this query, any help or lead to he

Hi,I need some assistance with Kudu, We have 2 tables that are consuming more space on our Kudu Cluster however, we could not find these tables under the database. Would need some guidance/assistance on a way to check where these tables are located under which Db. I can see the table in Kudu Web UI, however, when I try to query it via Impala I get the error: [My_server:21000] kwid> show create table ALARMS_COUNTERS_2021_11_01;Query: show create table ALARMS_COUNTERS_2021_11_01ERROR: AnalysisException: Table does not exist: kwid.ALARMS_COUNTERS_2021_11_01 If I run “show tables” against the DB I get a list of all the tables, but this table is not in the list. Thanks Wert