Q&A

I have 10 hive tables, few columns might be similar in few tables, I need to union them into one table. I was writing Union all statements.Now, these columns in these 10 tables might change . how to create the Union all query dynamically based on the tables.------------------------------------------------  CREATE VIEW comb_view ASSELECT col11, col12, col13,.....From(SELECT col11,col12,NULL col13 from tab1UNION ALLSELECT col11,NULL col12, col13 from tab2....) c-------------------------------------------------------ensuring the position and data type.

Hello,   IMPALAD SCM HEALTH Role health test bad IMPALAD UNEXPECTED EXITS Role health test bad IMPALAD QUERY MONITORING STATUS Role health test bad IMPALAD HOST HEALTH Role health test bad   Thanks

I installed and configured a fresh version of Apache Nifi 1.14.0 and configured security against my LDAP server. Everything looks fine. Now I copied my flow.xml.gz file from a different directory (also running 1.14) into the new directory. When I login, I can see the outline of all the process groups and processors, but all the names are missing, and I see that all my users are unauthorized to view anything or make any changes. Even my admin cannot do anything. What am I missing here? 

To whom it may concern,I have a cluster with Ambari 2.7.5.0 on it and HDP 3.1.5.  All the nodes in it are RHEL 7.9.I saw here:https://community.cloudera.com/t5/Support-Questions/java-update-Ambari/m-p/184506#M146626 ...someone posted instructions for updating Java on an older version of Ambari.  It boils down to putting the same JDK in the same path on all of your nodes, then doing a couple of ambari-server commands.Am I supposed to stop all of the individual applications (HDFS, YARN, ZooKeeper, etc.) before doing this, or does the 'ambari-server restart' on the Ambari server handle all that?  Is there an updated procedure for Ambari 2.7.5.0? Please respond when you can.Thanks! 

Hi Team, client production cluster 3 Weeks back suddenly down.That time they started the cluster services all services are came up but roll change happened and Standby name node is active and cluster is up and running.while starting the other name node(previous active name node) not coming up and strucked at "RPC wait" and in CM for that namenode status showing as 'BUSY'.if any one help me on this issue

I'm trying to sqoop some tables from Postgres to Hive and convert them to ORC while sqooping. 1. I used the below command.sqoop import --hcatalog-home /usr/hdp/current/hive-webhcat --connect jdbc:postgresql://<host>/<database> --username user --password pwd --table table --hcatalog-database default --hcatalog-table table  --hcatalog-storage-stanza "stored as orcfile" --create-hcatalog-table -m 1 I was getting the below error:ERROR tool.ImportTool: Import failed: org.apache.hive.hcatalog.common.HCatException : 2016 : Error operation not supported : Store into a transactional table default.agent from Pig/Mapreduce is not supported 2. I changed the sqoop to look like this.sqoop import --hcatalog-home /usr/hdp/current/hive-webhcat --connect jdbc:postgresql://<host>/<database> --username user --password pwd --table table --hcatalog-database default --hcatalog-table table  --hcatalog-storage-stanza "stored as orcfile tblproperties (\"transacti

Hi All, I am running distcp command which copies all the audit logs HDFS folder to another HDFS folder for further processing purpose .  The distcp command used to work fine till 2 weeks ago and started failing since last week .I checked detailed MR logs and understand that only particular file copy failed and other folder/files of audit logs like kafka,hive,nifi and hbase are copied . some specific files copy processing is failing. distcp command :hadoop distcp -filters $filter_file_loc ranger/audit /data/audit_logs/staging Distribution : Cloudera Data Platform version 7.1.7Please find the detail error messages . java.io.IOException: File copy failed: hdfs://namenode/ranger/audit/kafka/kafka/20210927/kafka_ranger_audit_svl.host.int.log --> hdfs://namenode/data/audit_logs/staging/audit/kafka/kafka/20210927/kafka_ranger_audit_svl.host.int.logCaused by: org.apache.hadoop.hdfs.CannotObtainBlockLengthException: Cannot obtain block length for LocatedBlo

Hi everyone,   I am new for cloudera software. I practice how to use hive and impala query from Hue but I got this error from Hue when I start query.       Could not connect to any of [('10.4.150.58', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.4.150.58', 10000)]",)       then I did some search and I know that because Hiveserver2 is down. I see log file and I got this.       12:18:07.280 PM ERROR HiveServer2 [main]: Error starting HiveServer2java.lang.Error: Max start attempts 5 exhausted at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1084) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1

We will often see the below error (either during fresh installation or after upgrade) that the jobs will intermittently fail (in my case case it failed in Hive): 2021-08-27 08:46:38,616 ERROR org.apache.curator.framework.imps.EnsembleTracker: [main-EventThread]: Invalid config event received: {server.1=********.mms.*.local:3181:4181:participant, version=0, server.3=**********.mms.*.local:3181:4181:participant, server.2=**********.mms.**.local:3181:4181:participant}2021-08-27 08:46:43,410 INFO org.apache.hadoop.hive.ql.txn.compactor.Cleaner: [Thread-15]: Cleaning based on min open txn id: 282021-08-27 08:46:43,744 WARN org.apache.zookeeper.ClientCnxn: [main-SendThread(*********.mms.***.local:2181)]: Session 0x1061dda12d30069 for server ********.mms.**.local/10.4.7.17:2181, unexpected error, closing socket connection and attempting reconnectjava.io.IOException: Packet len73388953 is out of range!at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java

I have a nifi instance running in kubernetes, I used the cetic/helm-nifi as the foundation although I did change some things to minimize the number of mounts to the server container. Anyways, I can view the UI using admin certificate created during setup, and when I visit the UI page I basically cannot use any of the buttons up top to create flows/etc.  I have attached pictures of my user policies and a picture of the top toolbar I'm talking about.  When I hover my mouse over any of the icons in the toolbar I get a circle with a line through it ('do not enter' sign).I assume I need additional permissions, but I'm not sure what to add (I'm guessing it will be an addition to the authorizations.xml file).My admin user policiesButtons up top are all non-functional (ring with a line through it when hovering)

Hello,We are seeing a few Impala queries running for more than 3 hours in Impala. Upon checking we found that the query is in exception state however on the Cloudera Impala GUI it still showing it as running. Would like to know if there is a way to kill such type of queries at Impala level.We have implemented Impala ACL and set the Queue Timeout to 5 minutes yet some queries keep running. Our applications connect to Impala via JDBC and we also have some users who use Hue to run queries. Any assistance/guidance is much appreciated. 

Hi, I am trying to create a dashboard  in CM with impala charts in order to see the maximum memory of impala and the one that my queries are using. I running tsquery for total_impala_admission_controller_local_backend_mem_reserved_across_impala_daemon_poolsandtotal_impala_admission_controller_local_backend_mem_usage_across_impala_daemon_pools But it seems that the charts are representing exactly the same thing. Why is this happens? 

Hi Team, I have been working on creating a new cluster "Cloudera Manager (Trial) 7.4.4". After installing hue I couldn't see hive editor and got to know we need hive on tez. So I installed hive on tex with one hiveserver and deleted hive server on hive. Now when I start the service my hiveserver in hive on tez goes down. Please help me with this. Error:[main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551] at org.apache.hive.service.server.HiveServer2.startHiveServer2(Hi

In Cloudera Manager., I could always see this Alert in health Tests Section - NameNode summary: xyz.abc.net (Availability: Unknown, Health: Good), pqr.abc.net (Availability: Unknown, Health: Good). This health test is bad because the Service Monitor did not find an active NameNode. I do not see any service Disruption and both the servers are up and running. Looking for help what need to be checked and why this heath test alert is persistent.  

HI I am learning Hadoop, I have tried word count python program I am not getting any error but it is stuck and nothing happen, cursor only comes in new line.I fired below commandHadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.6.0-mr1-cdh5.13.0.jar -file /home/cloudera/mapper.py /home/cloudera/reducer.py -mapper "python mapper.py" -reducer "python reducer.py -input /user/cloudera/PracticeSet/test.txt -output /user/cloudera/PracticeSet/

spark-shellFile "/bin/hdp-select", line 256print "ERROR: Invalid package - " + name^SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)?ls: cannot access /usr/hdp//hadoop/lib: No such file or directoryException in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxxat org.apache.spark.launcher.Main.main(Main.java:118)

I need to produce stats on cdsw resource usage by data scientists. See bellow: UserNumber of sessionsTotal CPUTotal data processed                    

I have a new Cloudera Cluster With services being used - Hdfs, Spark, Yarn, Hive, Kafka, Sentry , Oozie, Zookeeper, Streams Replication Manager, Streams Messaging Manager. I am working on implementing some processes in place about how to plan for OS Pathing for cluster nodes , When Services restart would be required and Sequence of steps and all. is there a defined documentation from cloudera in this  regard which i can refer to.Any hints please. 

  Examples : [ledapp3@leda-edge-u1-n01 ~]$ yarn logs -applicationId application_1616602302580_19604 >crash.logWARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.Permission denied: user=ledapp3, access=READ_EXECUTE, inode="/tmp/logs/hive/logs/application_1616602302580_19604":hive:hadoop:drwxrwx--- [calapr01@callisto-edge-u2-n01 ~]$ yarn logs -applicationId application_1623858219829_386030 | headWARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.Permission denied: user=calapr01, access=READ_EXECUTE, inode="/tmp/logs/hive/logs/application_1623858219829_386030":hive:hadoop:drwxrwx--- [gaiapr01@ganymede-edge-u2-n01 ~]$ yarn logs --applicationId application_1631641196769_7918 | headWARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.21/10/01 15:23:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm249Permission denied: user=gaiapr01, access=EXECUTE, inode="/tmp/logs/hive":hive:hadoop:drwx

We use the Cloudera ODBC Driver for Impala and Cloudera ODBC Driver for Hive inside a service. The service uses a DSN-less connection strings with UseKeytab to authenticate. We do not perform any ticket management, we leave this down to the ODBC driver. We have observed that connections to hive do not renew the TGT, and will reuse an existing service ticket (expected behaviour). However, Impala connections renew the TGT on every connection (thus destroying all service tickets..) Impala Service tickets never get reused and the ticket cache is always destroyed. This has to be a bug. To reproduce..Open ODBC Administrator and create a DSN connection to an Impala Server with kerberos keytab authentication.Test the connectionrun klist to show ticketsTest the connection againrun klist to show the tickets have been renewed

Hello all,I know this has been asked a lot of times before but I have not been able to resolve this issue using those fixes. I get this error twice or thrice every day. Someone please help.  3:44:45.647 PM WARN Configuration[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.exec.dynamic.partition.mode; Ignoring.3:44:45.647 PM WARN Configuration[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.compactor.initiator.on; Ignoring.3:44:45.648 PM WARN HiveConf[pool-8-thread-192]: HiveConf of name hive.sentry.conf.url does not exist3:44:45.648 PM WARN HiveConf[pool-8-thread-192]: HiveConf of name hive.enforce.bucketing does not exist Thanks,Shaban

I am attempting to enable Kerberos authentication for users, against an Active Directory based realm. I am following the model of having a MIT KDC to house Cloudera principals, and then establishing a Cross Realm Trust to the AD realm, to allow AD users to authenticate.   At a purely Kerberos level, this is working fine (see the example below), however when I attempt a cluster operation that requires Kerberos authentication, I see consistent and fairly general failures. I believe I have followed the AD integration docs, with the exception of the final part around configuring name translation. I don't think I have reached the stage in authentication where this is playing a role.   SITE.PRODUCT.COMPANY.ORG: Cluster KRB Realm SITE.COMPANY.ORG: Active Directory KRB Realm product.company.org: DNS domain of all cluster nodes auser: the test user account   If it has any relevance, the krb5.conf configuration for the AD Realm (and wider OS level AD integration) is managed by the

I am using Scala on data science workbench and I have file with sql stataement would like to make it dynamic sql by passing argument to parameters. I then would like to pass it  sqlContext.sql(string) . This is what I have tried but does not work.val FromDate = "2019-02-25"val sqlfile = fromFile("sql3.py").getLines.mkStringval result = sqlContext.sql(sqlfile)On the file I have:Select col1, col2 from table1 where transdate = '${FromDate}' Any help would be appreciated.  Thanks

Hi I have installed CDP with CDH 7.1 on my servers. We want to integrate our hue with hive. Earlier it used to show "Error loading databases" after that we uncommented following portion under interpreters section of notebook section in /etc/hue/conf/hue.ini file: [[[mysql]]]name = MySQLinterface=sqlalchemy## https://docs.sqlalchemy.org/en/latest/dialects/mysql.htmloptions='{"url": "mysql://hue:mypassword@myhostnameonwhichhuedbwascreatedandhueserviceisrunning:3306/hue"}'# ## options='{"url": "mysql://${USER}:${PASSWORD}@localhost:3306/hue"}'[[[hive]]]name=Hiveinterface=hiveserver2 it started showing the databases after making above configurations but its not showing me tables and giving me the following error as shown in below screenshot:'NoneType' object has no attribute 'settimeout'Hue db was created during installation by following this link, I gave path of this db in above config file:  https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/install_

I have some tables in hive and I want to find the size of each table through the metastore (MySQL).I am trying the below but it returns paratition_params, notification_log, sds etc. My tables are stored in TABLES.TBLS but when I am running the below query with from information_schema.TABLES.TBLS it returns this:You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.TBLSORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC LIMIT 0, 200' at line 5SELECT TABLE_NAME AS `Table`, ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024) AS `Size (MB)`FROM information_schema.TABLESORDER BY (DATA_LENGTH + INDEX_LENGTH)DESC;