Q&A

Hello,I've CDP 7.3.1 private cloud cluster (3 master, 1 manager, 1 edge & 5 slaves) . I'm wondering how can I secure a backup/restore point for it. Is it technically right and "supported by Cloudera" if I take a VM snapshots on machines level for master nodes ? Should it be cold (offline) backup (stop or pause the cluster operations) to ensure of consistencies between distributed services on master nodes? Thanks,Mahmoud

Hello,We are using Cloudera V5.1.1 with log4j v1.2.17, without option to upgrade Cloudera to later version at this moment. Does anyone has suggestion or solution how can we upgrade log4j component to latest version(2.16.0) on CDH V5.1.1?  Thanks 

Hello,I am getting an error after upgrade of CDH5.16 to CDP7.1.7. The logs show it is unable to connect to the KMS endpoint. First start a Spark session in sparkmagic then I run below example pyspark code: Starting Spark applicationID YARN Application ID Kind State Spark UI Driver log Current session? 23application_1639802810085_6070pysparkidleLinkLink✔ SparkSession available as 'spark'.###############sample pyspark code###################from pyspark.sql import SparkSessionspark = SparkSession.builder.master('local').getOrCreate()# load data from .csv file in HDFS# tips = spark.read.csv("/user/hive/warehouse/tips/", header=True, inferSchema=True)# OR load data from table in Hive metastoretips = spark.table('db1.table1')from pyspark.sql.functions import col, lit, mean# query using DataFrame API#tips \# query using SQLspark.sql("select name from db1.table1").show(3)spark.stop()  An error occurred while calling o85.showString.: java.io.IOException: java.lang.reflect.U

Dear Community,We are planning to install latest version (2.2.0) of the Apache Atlas, where can I find related versions of the core components of Atlas, like -HBase, Solr, Graph Engine.This is mainly to evaluate Log4j vulnerability 

I have a MiNiFi agent that tails from a file and sends it to a NiFi Cluster. After a while, The state directory takes 27 Gigabytes of the disk and the content_repository only takes 34MB of the disk. Any idea about this problem?

Hi, I'm new to Minifi and just got minifi cpp running with the listed examples from the github repo. In the next step I want to run my own config.yml files. Unfortunately the minifi cpp repo doesn't explain the tranformation process from Nifi to Minifi. However, on the minifi website I read that the minifi toolkit is used for the transformation process from the template xml to yml-file. I downloaded the latest toolkit version (v1.14.0) and exportet a simple flow template as xml (Nifi-version is 1.12.1). When I try to convert the xml-file using the toolkit I get the folowwing error: [user@ip-xxx bin]$ ./config.sh transform input.xml outputfileconfig.sh: JAVA_HOME not set; results may varyJava home:MiNiFi Toolkit home: /home/user/minifi-toolkit-1.14.0Error reading template. (javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.- with linked exception:[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory

Hi folks, We've just install CDP Public Cloud over Azure and we are working with data hubs cluster. We have a brand new environment and data hubs. We are unable to access to Nifi UI with any user except the account owner. The users have privileges in the environment(EnvironmentUser) and also in the data hub (DHUser) What should I do next? is this related with ranger? Thanks in advance

Hello.I want to disable logging for some of the policies in Ranger (Ranger 1.2.0, HDP-3.1.4.0-315) I edit the policies to switch off the Audit Logging button and when I save this configuration in Ranger Audit I see that nothing changes (all the logs from the policies that I disabled logging are visible).Why can this be so?

Hi,what changes we need to be made if we need to change the ip address of the master and worker servers? Thanks

I added anew worker to the CDSW cluster and the status of this worker is false,In the logs I gut this message:Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")  

Hello,We need to export Cloudera Manager audit logs (not to be confused with Ranger audit logs) to some external system.Previously (in CDH) they were Navigator logs, but these are not present anymore.Is there any solution for exporting these logs?I know they are available on /cmf/history endpoint, but maybe there is some more convenient way of reading them? Best regards,Aleksander

I have a csv data which I have to load in impala/hive. does anyone knows how to remove the double quotation mark in the output? Here is my sample create table scripts. CREATE EXTERNAL TABLE schema.table (   id int,   name STRING,   desc STRING,   desc1 STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' WITH SERDEPROPERTIES ('field.delim'='|', 'serialization.format'='|') STORED AS TEXTFILE LOCATION '/user/folder/schema/table'

HI We are currently trying to restart the HDFS name nodes in our cluster (HA). It seems that we are not able to start them. The process of restarting the node takes a long time until it finally fail with the following output on AMBARI: Operation failed: Call From XX-XXX-XX-XXXX.XXXXX.XX/XX.X.XX.XX to XX-XXX-XX-XXXX.XXXXX.XX:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused2021-12-16 17:02:46,616 - call returned (255, '21/12/16 17:02:46 INFO ipc.Client: Retrying connect to server: XX-XXX-XX-XXXX.XXXXX.XX/XX.X.XX.XX:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From XX-XXX-XX-XXXX.XXXXX.XX/XX.X.XX.XX to XX-XXX-XX-XXXX.XXXXX.XX:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefuse

Hello, After upgrading to CM 7.4.4/CR 7.1.7 the Activity Monitor never starts. It is always down and cannot be started after multiple attempts. It gives error messages below any thoughts?Command (Start this Activity Monitor (17683)) has failedSupervisor returned FATAL. Please check the role log file, stderr, or stdout.The health test result for ACTIVITY_MONITOR_SCM_HEALTH has become bad: This role's process failed to start.STDOUT messages:13:06:12.204 [main] ERROR org.hibernate.engine.jdbc.batch.internal.BatchingBatch - HHH000315: Exception executing batch [java.sql.BatchUpdateException: Data truncation: Out of range value for column 'METRIC_ID' at row 1], SQL: insert into CMON_METRIC_INFO (NAME, METRIC_ID) values (?, ?)13:06:12.209 [main] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Data truncation: Out of range value for column 'METRIC_ID' at row 1Sat Dec 18 13:06:17 EST 2021JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64CONF_DIR=/var/run/cloud

I need  password when I downloaded new version(Ambari 2.7.5 and HDP 3.1.5).I read the document from here(https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/access_ambari_paywall.html),and then I opened url(https://my.cloudera.com) in the document and did not know how to get the user and password.So, aren't they open source anymore?

Cannot display the number of topics and information about each topic when using cmd (zookeeper state changed (SyncConnected)) 

Hi Team, I am working on a mule integration project and so need to access the Impala database. I am using a generic database connection with the ImpalaJDBC42.jar as the maven dependency. Mule app uses maven plugin to build the app first even before trying to test database connections.I am getting the following error (abridged version of the stacktrace):Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at Impala:ImpalaJDBC42:jar:2.6.24.1029.....Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at Impala:ImpalaJDBC42:jar:2.6.24.1029Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for Impala:ImpalaJDBC42:jar:2.6.24.102.....Caused by: org.apache.maven.model.building.ModelBuildingException: 3 problems were encountered while building the effective model for Impala:ImpalaJDBC${env.JDBC_V}:${env.MAJOR_V}.${env.MINO

Hi experts,Can someone please explain the difference between volumes and folders in hadoop? Thanks,

Hi I have some issue with my queries in Hive, they are very simple queries.  One day start to fail and this is the error : Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Could not find status of job:job_1639152705224_0018  INFO : Kill Command = /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hadoop/bin/hadoop job -kill job_1639152705224_0018INFO : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0INFO : 2021-12-10 17:35:35,913 Stage-1 map = 0%, reduce = 0%INFO : 2021-12-10 17:35:43,176 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 7.34 secERROR : Ended Job = job_1639152705224_0018 with exception 'java.io.IOException(Could not find status of job:job_1639152705224_0018)'java.io.IOException: Could not find status of job:job_1639152705224_0018 at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.ja

I have a Hive server in my cluster running a MySQL instance.  The logfile /var/run/mysqld/mysqld.log is getting really large and I need to figure out the proper way to prune it. The server it's running on is CentOS 7.  Can I just 'cat /dev/null > mysqld.log' to clear it out, or are there other considerations?  Can I get away with simply stopping the MySQL instance briefly to delete or 'zero out' the file, or do I need to stop all the services on that Hive master node in Ambari? Please advise if you can.Thanks,Mike

I'm using CDH version 5.13, but I can't see the bdr function! Help!!!

hello all, We are using Nifi to do an incremental load from SQL Server using QueryDatabaseTableRecord where max-value column is updated_at column with a Datetime2 data type. The source tables get updated in milliseconds. We noticed that we are losing few events doing with each QueryDatabaseTableRecord run. As a work-around we created a custom processor that subtracts few seconds from the max-value in the state fetching overlapping events and then we handle any duplicates later on in the pipeline and it seems to solve the issue. However, this is not optimal. We'd like to know what's causing the skips. Our assumption so far is that timestamp datatype accuracy is causing the skips in the incremental load however we're not sure.   

I get a file with multiple dates in the following format: 2021-08-25T12:34:48.984704 which is in micro format. How could I change these dates on the nifi to stay in millis? I tried the following: Apache Nifi Expression Language - toDate formatting formatting but I was not successful in the configuration. I believe that the answer is cohort, but I am not managing to make it work.

We need to update our LDAP configuration because our certificate is going to expire; we have a test ldaps server set up with the new certificate.  From within CDH->Administration->Settings, I pointed the "LDAP URL" to the new server. simple bind failed: ldapsdev.{obscured domain}:3269; nested exception is javax.naming.CommunicationException: simple bind failed: ldapsdev.{obscureddomain}:3269 [Root exception is javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure]  Under the assumption that a cert needed to be added, I tried to figure out how, but could find zero documentation other than this:https://community.cloudera.com/t5/Community-Articles/Steps-to-setup-Atlas-with-Ldaps-SSL/ta-p/247365That relates to Atlas, which we don't use, but seemed right.  I downloaded "ldapsdev-ca.crt" from the ldaps server: echo -n | openssl s_client -connect ldapsdev.{obscureddomain}:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >

Hello everyone in this day I tried to convert CSV  file to xlsx file using Apache nifi but i couldn't anyone knows? or any idea on how to get NIFI flow output as .xlsx file