Solved

Our customer looking to get logs of all jobs ran in a particular cluster .How to gather such information ?can someone help on this.

Dears,   I have a requirement to enable all backups and restoration for all services of CDH and CM and Management services and their meta data db's  and configuration of CDH clod & hot clusters. Kindly support with needful details.

Hi experts,I just wanted to confirm my understanding or help me better understand the yarn local and log dirs.So my understanding is that yarn will download the data locally to a filesystem so that it is more easily accessible when a job is run as well as logs for that particular application. I believe these are temporary files as they will be stored in HDFS after the job completes. [root@test01 log]# ll /hadoop/yarn/total 0drwxr-xr-x. 6 yarn hadoop 78 Oct 13 08:09 localdrwxrwxr-x. 8 yarn hadoop 239 Oct 13 08:09 logCan someone please help confirm my understanding or help me better understand this concept?Also what is usually best practice in regards to mounting these directories onto a local filesystem or onto another hard drive or can I have this directory share a hard drive with one of the datanode directories? Any help is much appreciated. Thanks,

I installed and configured a fresh version of Apache Nifi 1.14.0 and configured security against my LDAP server. Everything looks fine. Now I copied my flow.xml.gz file from a different directory (also running 1.14) into the new directory. When I login, I can see the outline of all the process groups and processors, but all the names are missing, and I see that all my users are unauthorized to view anything or make any changes. Even my admin cannot do anything. What am I missing here? 

To whom it may concern,I have a cluster with Ambari 2.7.5.0 on it and HDP 3.1.5.  All the nodes in it are RHEL 7.9.I saw here:https://community.cloudera.com/t5/Support-Questions/java-update-Ambari/m-p/184506#M146626 ...someone posted instructions for updating Java on an older version of Ambari.  It boils down to putting the same JDK in the same path on all of your nodes, then doing a couple of ambari-server commands.Am I supposed to stop all of the individual applications (HDFS, YARN, ZooKeeper, etc.) before doing this, or does the 'ambari-server restart' on the Ambari server handle all that?  Is there an updated procedure for Ambari 2.7.5.0? Please respond when you can.Thanks! 

Hi Team, client production cluster 3 Weeks back suddenly down.That time they started the cluster services all services are came up but roll change happened and Standby name node is active and cluster is up and running.while starting the other name node(previous active name node) not coming up and strucked at "RPC wait" and in CM for that namenode status showing as 'BUSY'.if any one help me on this issue

Hi everyone,   I am new for cloudera software. I practice how to use hive and impala query from Hue but I got this error from Hue when I start query.       Could not connect to any of [('10.4.150.58', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.4.150.58', 10000)]",)       then I did some search and I know that because Hiveserver2 is down. I see log file and I got this.       12:18:07.280 PM ERROR HiveServer2 [main]: Error starting HiveServer2java.lang.Error: Max start attempts 5 exhausted at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1084) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1

We will often see the below error (either during fresh installation or after upgrade) that the jobs will intermittently fail (in my case case it failed in Hive): 2021-08-27 08:46:38,616 ERROR org.apache.curator.framework.imps.EnsembleTracker: [main-EventThread]: Invalid config event received: {server.1=********.mms.*.local:3181:4181:participant, version=0, server.3=**********.mms.*.local:3181:4181:participant, server.2=**********.mms.**.local:3181:4181:participant}2021-08-27 08:46:43,410 INFO org.apache.hadoop.hive.ql.txn.compactor.Cleaner: [Thread-15]: Cleaning based on min open txn id: 282021-08-27 08:46:43,744 WARN org.apache.zookeeper.ClientCnxn: [main-SendThread(*********.mms.***.local:2181)]: Session 0x1061dda12d30069 for server ********.mms.**.local/10.4.7.17:2181, unexpected error, closing socket connection and attempting reconnectjava.io.IOException: Packet len73388953 is out of range!at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java

I have a nifi instance running in kubernetes, I used the cetic/helm-nifi as the foundation although I did change some things to minimize the number of mounts to the server container. Anyways, I can view the UI using admin certificate created during setup, and when I visit the UI page I basically cannot use any of the buttons up top to create flows/etc.  I have attached pictures of my user policies and a picture of the top toolbar I'm talking about.  When I hover my mouse over any of the icons in the toolbar I get a circle with a line through it ('do not enter' sign).I assume I need additional permissions, but I'm not sure what to add (I'm guessing it will be an addition to the authorizations.xml file).My admin user policiesButtons up top are all non-functional (ring with a line through it when hovering)

Hello,We are seeing a few Impala queries running for more than 3 hours in Impala. Upon checking we found that the query is in exception state however on the Cloudera Impala GUI it still showing it as running. Would like to know if there is a way to kill such type of queries at Impala level.We have implemented Impala ACL and set the Queue Timeout to 5 minutes yet some queries keep running. Our applications connect to Impala via JDBC and we also have some users who use Hue to run queries. Any assistance/guidance is much appreciated. 

Hi, I am trying to create a dashboard  in CM with impala charts in order to see the maximum memory of impala and the one that my queries are using. I running tsquery for total_impala_admission_controller_local_backend_mem_reserved_across_impala_daemon_poolsandtotal_impala_admission_controller_local_backend_mem_usage_across_impala_daemon_pools But it seems that the charts are representing exactly the same thing. Why is this happens? 

Hi Team, I have been working on creating a new cluster "Cloudera Manager (Trial) 7.4.4". After installing hue I couldn't see hive editor and got to know we need hive on tez. So I installed hive on tex with one hiveserver and deleted hive server on hive. Now when I start the service my hiveserver in hive on tez goes down. Please help me with this. Error:[main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.startHiveServer2(Hi

In Cloudera Manager., I could always see this Alert in health Tests Section - NameNode summary: xyz.abc.net (Availability: Unknown, Health: Good), pqr.abc.net (Availability: Unknown, Health: Good). This health test is bad because the Service Monitor did not find an active NameNode. I do not see any service Disruption and both the servers are up and running. Looking for help what need to be checked and why this heath test alert is persistent.  

spark-shellFile "/bin/hdp-select", line 256print "ERROR: Invalid package - " + name^SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)?ls: cannot access /usr/hdp//hadoop/lib: No such file or directoryException in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxxat org.apache.spark.launcher.Main.main(Main.java:118)

I need to produce stats on cdsw resource usage by data scientists. See bellow: UserNumber of sessionsTotal CPUTotal data processed                    

I have a new Cloudera Cluster With services being used - Hdfs, Spark, Yarn, Hive, Kafka, Sentry , Oozie, Zookeeper, Streams Replication Manager, Streams Messaging Manager. I am working on implementing some processes in place about how to plan for OS Pathing for cluster nodes , When Services restart would be required and Sequence of steps and all. is there a defined documentation from cloudera in this  regard which i can refer to.Any hints please. 

  Examples : [ledapp3@leda-edge-u1-n01 ~]$ yarn logs -applicationId application_1616602302580_19604 >crash.logWARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.Permission denied: user=ledapp3, access=READ_EXECUTE, inode="/tmp/logs/hive/logs/application_1616602302580_19604":hive:hadoop:drwxrwx--- [calapr01@callisto-edge-u2-n01 ~]$ yarn logs -applicationId application_1623858219829_386030 | headWARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.Permission denied: user=calapr01, access=READ_EXECUTE, inode="/tmp/logs/hive/logs/application_1623858219829_386030":hive:hadoop:drwxrwx--- [gaiapr01@ganymede-edge-u2-n01 ~]$ yarn logs --applicationId application_1631641196769_7918 | headWARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.21/10/01 15:23:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm249Permission denied: user=gaiapr01, access=EXECUTE, inode="/tmp/logs/hive":hive:hadoop:drwx

Hello all,I know this has been asked a lot of times before but I have not been able to resolve this issue using those fixes. I get this error twice or thrice every day. Someone please help.  3:44:45.647 PM WARN Configuration[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.exec.dynamic.partition.mode; Ignoring.3:44:45.647 PM WARN Configuration[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.compactor.initiator.on; Ignoring.3:44:45.648 PM WARN HiveConf[pool-8-thread-192]: HiveConf of name hive.sentry.conf.url does not exist3:44:45.648 PM WARN HiveConf[pool-8-thread-192]: HiveConf of name hive.enforce.bucketing does not exist Thanks,Shaban

I am attempting to enable Kerberos authentication for users, against an Active Directory based realm. I am following the model of having a MIT KDC to house Cloudera principals, and then establishing a Cross Realm Trust to the AD realm, to allow AD users to authenticate.   At a purely Kerberos level, this is working fine (see the example below), however when I attempt a cluster operation that requires Kerberos authentication, I see consistent and fairly general failures. I believe I have followed the AD integration docs, with the exception of the final part around configuring name translation. I don't think I have reached the stage in authentication where this is playing a role.   SITE.PRODUCT.COMPANY.ORG: Cluster KRB Realm SITE.COMPANY.ORG: Active Directory KRB Realm product.company.org: DNS domain of all cluster nodes auser: the test user account   If it has any relevance, the krb5.conf configuration for the AD Realm (and wider OS level AD integration) is managed by the

I am using Scala on data science workbench and I have file with sql stataement would like to make it dynamic sql by passing argument to parameters. I then would like to pass it  sqlContext.sql(string) . This is what I have tried but does not work.val FromDate = "2019-02-25"val sqlfile = fromFile("sql3.py").getLines.mkStringval result = sqlContext.sql(sqlfile)On the file I have:Select col1, col2 from table1 where transdate = '${FromDate}' Any help would be appreciated.  Thanks

Hi I have installed CDP with CDH 7.1 on my servers. We want to integrate our hue with hive. Earlier it used to show "Error loading databases" after that we uncommented following portion under interpreters section of notebook section in /etc/hue/conf/hue.ini file: [[[mysql]]]name = MySQLinterface=sqlalchemy## https://docs.sqlalchemy.org/en/latest/dialects/mysql.htmloptions='{"url": "mysql://hue:mypassword@myhostnameonwhichhuedbwascreatedandhueserviceisrunning:3306/hue"}'# ## options='{"url": "mysql://${USER}:${PASSWORD}@localhost:3306/hue"}'[[[hive]]]name=Hiveinterface=hiveserver2 it started showing the databases after making above configurations but its not showing me tables and giving me the following error as shown in below screenshot:'NoneType' object has no attribute 'settimeout'Hue db was created during installation by following this link, I gave path of this db in above config file:  https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/install_

Hello, I'm trying to use a ODBC connexion with SAP BI 4.2 app, on linux environment. Here are my settings :  --------- odbc.ini file -----------------[ODBC Data Sources]Impala=Simba Impala ODBC Driver (64-bit)[Impala_TEST]Description=Simba Impala ODBC Driver (64-bit)Driver=/data/BO/sap_bobj/enterprise_xi40/linux_x64/odbc/simbaimpala/2.0/lib/libimpalaodbc_sb64.soHiveServerType=2ServiceDiscoveryMode=0Host=portalgps.idf.fr.gin.intraorangePort=443schema=defaultAuthMech=4KrbFQDN=_HOSTUID=hiveThriftTransport=2HttpPath=/knox/cdp-proxy-api/hiveServiceDiscoveryMode=0AllowSelfSignedServerCert=1UseNativeQuery=1SSL=1 AllowSelfSignedCerts=1CAIssuedCertNamesMismatch=1TrustedCerts=/data/BO/sap_bobj/enterprise_xi40/linux_x64/odbc/simbaimpala/2.0/lib/cacerts.pemTwoWaySSL=0   --------- odbcinst.ini -----------------[ODBC Drivers]Simba Impala ODBC Driver 64-bit=InstalledImpala_TEST=Installed[Simba Impala ODBC Driver 64-bit]Description=Simba Impala ODBC Driver (64-b

Hi, I am using nifi 1.14. My entire workflow has been crashed and suddenly Nifi is showing blank canvas. Is there any safe way to recover all the jobs? Please provide step by step solution. Any help in this issue will be greatly appreciated.Thank You!

Hello, everyone,I'm just starting to get to grips with NiFi and I'm developing a flow that involves taking csv files from an SFTP server and transferring them locally.The task itself seems very simple to me, I found that a viable solution would be to use ListSFTP and FetchSFTP for the purpose. My problem lies in the connection to the SFTP server and in particular in passing the 'Private Key Path' parameter. My private key is stored in a .ppk file on the virtual machine I am using, which uses UNC paths. The moment I pass the UNC path to the processor with the pointing to the .ppk file, the processor returns an alert, telling me that the file does not exist. I can't figure out if the problem is in the way I passed the path to the processor or if it's something else.The path I passed to the processor looks like this:\\computer_name\View_UserData$\user_name\Desktop\private_key.ppk In addition, I have set the Port parameter to 22, and the connection also requires a Password.Can anyone

I am working off of a 3-node NiFi cluster and that is kicked off by a GenerateFlowfileProcessor run on the primary node, performs some NiFi processing, and then writes the files to the server that I will then run an ExecuteStreamCommand Python script on. The problem I’m running into is I can’t figure out a way to ensure that the processors picking up the first output are run on the same node as the processors that produced the first output. What is the best way to handle producing files that can be accessed by all nodes?Is there a way to specify the node for a process will be run on? (using “run on primary” is not working as the primary node cycles over the process)