Hello, How do I check the location where Kerberos is writing logs? I checked the location which is mentioned in krb5.conf (default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log) however the log files mentioned in this location are empty. Am I checking an incorrect location?   Secondly logs on Hue Server & KT renewer is not getting updated / current.   Any help/guidance is appreciated. ... View more

Hi, Wanted clarification/guidance related to finalize process for HDFS. Could we keep using the cluster while the finalize upgrade process is ongoing e.g., data is being written to hdfs while finalize process continues in the background OR would we need downtime for the same?   Thanks Wert ... View more

Hi, I need some assistance with Kudu, We have 2 tables that are consuming more space on our Kudu Cluster however, we could not find these tables under the database. Would need some guidance/assistance on a way to check where these tables are located under which Db.   I can see the table in Kudu Web UI, however, when I try to query it via Impala I get the error:   [My_server:21000] kwid> show create table ALARMS_COUNTERS_2021_11_01; Query: show create table ALARMS_COUNTERS_2021_11_01 ERROR: AnalysisException: Table does not exist: kwid.ALARMS_COUNTERS_2021_11_01   If I run “show tables” against the DB I get a list of all the tables, but this table is not in the list.   Thanks  Wert ... View more

Hello, We are observing error in Impala Catalouge Logs as below EI1101 14:52:24.768759 8438 jni-util.cc:308] org.apache.impala.common.InternalException: Error updating the catalog due to lock contention. at org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:3474) at org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:308) I1101 14:52:24.768983 8438 status.cc:125] InternalException: Error updating the catalog due to lock contention. @ 0x9812da @ 0xd37e2f @ 0x970728 @ 0x960e7f @ 0xa1f263 @ 0xa1752f @ 0x948d4a @ 0xb420c9 @ 0xb39fc9 @ 0xb3adc2 @ 0xdab24f @ 0xdaba4a @ 0x13346fa @ 0x7fb75bf7cea5 @ 0x7fb75bca59fd   CM / CDH - 5.16.2 Appericate any assistacne / guidance in fixing this issue.   Thanks  Wert   ... View more

Hello,   Wanted to know what features/components in CM/CDH 6.3.3 would stop working once the enterprise license is expired.   Thanks  Wert ... View more

Hello, I would like some guidance/ information on data distribution in Kudu T-Servers. We have Kudu cluster of 3 Masters and 9 T-Servers (each t-server has storage of 1TB). We are noticing that space in some t-server is getting consumed rapidly whereas in other its not that much being consumed. Would like to know why this is happening and is there any way this can be overcome, so that data can be distributed evenly across of 9 t-servers. Kudu 1.7.0-cdh5.16.2/ CM 5.16.2   Appreciate any assistance in this regard.   Thanks Wert   ... View more

Hello,   How do we balance data which are stored on individual disk on a particular datanode, we have 5 disks on a single node and one of the disk is 90% full, running balancer is not fixing the issue. Would like to get some suggestions/ comments to fix this issue. I was going through this article and it says it is not possible to balance disks within a single node, so what other options can we use to fix this until we upgrade to CDH 6.3 (https://community.cloudera.com/t5/Community-Articles/HDFS-Balancer-Balancing-Data-Between-Disks-on-a-DataNode/ta-p/244650)   CM & CDH - 5.16.3   Appreciate all inputs.   Thanks Wert ... View more

Hello, I am trying to execute hdfs fsck command and am getting below error:   [root@server1 root]# hdfs fsck / > /home/test/fsck_output_2-11-21 Connecting to namenode via http://server1.com:50070/fsck?ugi=hdfs&path=%2F Exception in thread "main" java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:363) at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:72) at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:161) at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:158) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:406)   Thanks wert ... View more

Hello, We were getting ‘DatabaseError: ORA-01017: invalid username/password; logon denied’ post our db upgrade, to fix that we have reset the hue password, which worked and that issue got fixed. Unfortunately one of our guys deleted a couple of logs from /var/log/hue. We have recreated runcpserver.log with the necessary permission, and restart Hue server, yet we fail to see the file being populated (file is still empty).   Request assistance / guidance in fixing this issue.   Thanks Wert ... View more

Hello, We have got our Domain renamed from ABC to XYZ.com. Since then neither Cloudera Manager UI nor any other service UI's are accessible. As per my understanding, we would need to update the /etc/hosts file of each node in the cluster to reflect the new Domain name. however it would be great if anyone could advise on the below 2 things: 1. Do we really need to get /etc/host/file updated, OR is it taken care by Cloudera 2. Where all do we need to update the hostnames with new Domain name to have a fully functional cluster back to operational status Currently if I manually enter the complete address (ip+new doamin name) I can login to Cloudera Manager UI but from Cloudera Manager when I try to open any service UI's like NN UI it fails.   Any help is much appreciated   Regards Wert ... View more

Hello, Would like some assistance/guidance on Kerberos. Our domain name has changed and since then our applications are unable to connect to Hadoop cluster. We are using MIT Kerberos.   Regards Wert ... View more

Hello,   I was using Cloudera Enterprise 5.16 (Trial) which is now downgraded to Cloudera Express, since then I am facing issues starting Reports Manager. I am using the default embedded database.   CM / CDH - 5.16.2     2020-09-06 14:14:29,706 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist. 2020-09-06 14:14:29,711 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists. 2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 4100. 2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 4100 2020-09-06 14:14:29,725 INFO com.cloudera.enterprise.ssl.SSLFactory: Using default java truststore for verification of server certificates in HTTPS communication. 2020-09-06 14:14:29,761 WARN com.cloudera.cmf.BasicScmProxy: Exception while getting fetch configDefaults hash: none java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:276) at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:596) at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:686) at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64) at com.cloudera.headlamp.HeadlampServer.<init>(HeadlampServer.java:143) at com.cloudera.headlamp.HeadlampServer.main(HeadlampServer.java:250) 2020-09-06 14:14:29,772 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 1 tries, sleeping for 2 secs 2020-09-06 14:14:31,793 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 2 tries, sleeping for 2 secs 2020-09-06 14:14:33,794 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 3 tries, sleeping for 2 secs 2020-09-06 14:14:35,795 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 4 tries, sleeping for 2 secs 2020-09-06 14:14:37,797 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 5 tries, sleeping for 2 secs 2020-09-06 14:14:39,797 ERROR com.cloudera.headlamp.HeadlampServer: Could not fetch descriptor after 5 tries, exiting.       Appreciate any help.   ... View more