Member since
07-30-2020
156
Posts
25
Kudos Received
41
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
199 | 01-11-2023 07:59 AM | |
498 | 12-29-2022 12:04 AM | |
180 | 12-28-2022 06:17 AM | |
305 | 12-07-2022 10:31 PM | |
456 | 12-06-2022 12:49 AM |
07-25-2022
04:49 AM
Hi Hanni, Are you able to view the content from the commndline? When accessing the files from Web Ui, the default user as per hadoop.http.staticuser.user is drwho. So make sure this user has the necessary permissions for the file. As the file has read permission for others, do check the Namenode logs for any warnings. Also, try using a different browser and check if above doesn't help.
... View more
07-24-2022
12:49 AM
1 Kudo
Hi @Meepoljd As per the description of the issue, I believe you are hitting ZOOKEEPER-1622 . Cloudera has already released TSB-2022-577 for this. As mentioned in the TSB, the session IDs should be unique but due to this bug, two clients might use the same session ID and hence causing the unexpected session termination.
... View more
07-22-2022
02:43 AM
1 Kudo
Hi, A hole in region chain most probably indicates there are some regions which are not yet online and hence creates a hole. # cat hbck.report | grep "not deployed on any region server" If you see regions in the above command output, you will need to assign them using hbase shell.
... View more
07-22-2022
01:44 AM
1 Kudo
Hello @loridigia You can try to assign the region from hbase shell. > assign ' 28dd7c81713c9347e8dfe4e6993b1ec7' If you can attach the below command output (with valid ticket ), we can check which all regions are offiline or in transition. # hbase hbck -details
... View more
07-21-2022
07:37 AM
As per the stack trace, it seems to be an issue with the JDK. Do check HIVE-21237 HIVE-21584
... View more
07-21-2022
07:24 AM
1 Kudo
Hello @loridigia , It seems due to the outage there would be multiple ServerCrashProcedures created for the Regionservers. The Dead region severs with same names are different instances of the Region servers with a different epoch timestamp. As the Hbase Master was also down, it might be possible that it was not able to process the expiration of the Region servers. You might see some Crash procedures waiting to be finished under "Procedures & Locks" section of the Active Hbase Master Web UI. As you have already solved this issue in the past involving zookeeper. I guess you can try this : 1. Stop Hbase 2. Login to zookeeper using #hbase zkcli ( with a valid hbase ticket ) 3. Delete the /hbase-secure znode. rmr /hbase-secure 4. Sideline the entries under HDFS dir. hdfs dfs -mv /hbase/ MasterProcWALs/* /tmp. ( Not sure if this was done earlier ) 5. Start Hbase
... View more
07-18-2022
02:00 AM
Hello @KPG1 , The time taken to mark a datanode as stale is give by dfs.namenode.stale.datanode.interval, with a default of 30 seconds. If this is happening with a specific Datanode, you can check if there is any network issues between the Datanode and the Namenode or if the Datanode has any JVM pauses reported by checking the Datanode logs. As a bandaid, you can bump up the above parameter till the underlying problem is solved.
... View more
07-13-2022
10:55 AM
Hello, Based on the above test, I guess you are hitting HBASE-21852 which is still unresolved in the upstream.
... View more
07-13-2022
07:35 AM
Hello, The encoded value for \x would be %5Cx. So can you try using it in the url. Is it connecting via Knox? Also , do upload the curl command output.
... View more
07-04-2022
06:21 AM
2 Kudos
@stale , It looks like a mismatch in the encryption types in your krb5.conf and the AD is causing this. Do check the below 2 Cloudera articles to see if that helps resolving this issue. https://my.cloudera.com/knowledge/ERRORquotCaused-by-GSSException-Failure-unspecified-at-GSS-API?id=272836 https://my.cloudera.com/knowledge/ErrorquotCaused-by-Failure-unspecified-at-GSS-API-level?id=273436
... View more
06-30-2022
03:41 AM
Hello @Grumash , I believe user=cdp_svc_fc_03 is the spark user which no longer exists. So when you are trying to move the file into the trash folder in the home dir, its failing to create the home dir. You need to create the home dir as the super user (hdfs), then chown it to cdp_svc_fc_03, then it should work.
... View more
06-30-2022
03:30 AM
1 Kudo
Hello, There is no one Click solution in CDP to disable kerberos. As you pointed out, it is not recommended to disable kerberos once it's configured. Without kerberos, Ranger and other services might not work properly as Kerberos is the core security in CDP. You can try to follow the below community post to check if that helps. https://community.cloudera.com/t5/Support-Questions/Disabling-Kerberos/m-p/19934#M38077
... View more
06-09-2022
10:33 AM
Hi @KiranMagdum , The last failure noticed is on April 28th. So if you are not seeing any permission/access issue for the disk /disk3/dfs/dn on Datanode log of 10.204.8.11, can you try to restart this datanode role and check if the Namenode UI still prints the Volume as failed.
... View more
06-09-2022
12:58 AM
Hi @Jessica_cisco, The hbase:meta ( system table ) is not online and thus the Hbase Master has not come out of initialisation phase. We will need to assign the region for the hbase:meta table, as this table contains the mapping regarding which region is hosted on a specific Region server. The workflow is failing as this hbase:meta table is not assigned/online. We need to use hbck2 jar ( we get this via support ticket) to assign this region for which the best way is to open a Cloudera Support ticket. https://docs.cloudera.com/runtime/7.2.10/troubleshooting-hbase/topics/hbase_fix_issues_hbck.html Else, ( Doesn't work everytime ) You can try to restart the Region server data-02.novalocal followed by a restart of the Hbase Master to see if the Master is able to assign the meta table. Regards, Robin
... View more
06-08-2022
03:16 AM
Hi @enirys , You will need to add the host entries in the DNS record if freeipa is used to manage the DNS. You can compare the host entries from the other working Datanode in freeipa. Every node in a Data Lake, Data Hub, and a CDP data service should be configured to look up the FreeIPA DNS service for name resolution within the cluster. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/adding-host-entry
... View more
06-07-2022
11:25 AM
Hi @enirys , It looks like a DNS resolution issue. Could you check if this gets resolved by following this article https://my.cloudera.com/knowledge/ERROR-quot-is-not-authorized-for-protocol-interface?id=304462
... View more
06-07-2022
11:11 AM
1 Kudo
Hi @Jessica_cisco , As per the screenshot, it reports that the Region server on this host has failed to start. You can login to that host and confirm if there is a Region server process running. #ps -ef | grep regionserver If you don't see any process, try to restart this Region server from CM and if it still fails, please check the stderr and the role log of this Region server for more clues.
... View more
06-07-2022
01:30 AM
Hello @Jessica_cisco , You can check if cloudera-manager.repo is present on this host under /etc/yum.repos.d/. If not, copy this repo from a working node. If you run the below command on this host, it should show you the repo from where it will download the agent package. # yum whatprovides cloudera-manager-agent cloudera-manager-agent-7.6.1-24046616.el7.x86_64 : The Cloudera Manager Agent Repo : @cloudera-manager Once the above is confirmed, You can use the below doc for instructions. https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/installation/topics/cdpdc-manually-install-cm-agent-packages.html
... View more
06-07-2022
12:51 AM
1 Kudo
Hello, Try to install the agent package on this host and let me know if that solves the issue. Make a copy of your /etc/cloudera-scm-agent/config.ini. Uninstall the cloudera-manager-agent package Install the cloudera-manager-agent package Copy the /etc/cloudera-scm-agent/config.ini back Start the cloudera-scm-agent service
... View more
06-06-2022
06:51 AM
1 Kudo
Hi, Can you check if agent package is indeed present on this host. You can compare it with a working host. # rpm -qa | grep cloudera
... View more
06-06-2022
03:57 AM
Hi, Do you see any errors in the Cloudera-scm-agent logs? If the agent looses connection to the CM Server, it reports the host as unhealthy. Have you tried restarting the Cloudera-scm-agent on this host and check if that helps.
... View more
06-03-2022
02:53 AM
It could be possible that the Datanode is still in decommissioning state. Can you start to Stop that Datanode from the UI and then try to exit the Maintenance mode and check.
... View more
06-01-2022
04:45 AM
Cell TTL is something that is defined at insertion time. So for an already existing data, you can perform a put of the same cell, which then will have the exact cell value on a new Hfile with the new TTL for the cell.
... View more
05-31-2022
10:27 AM
As HDFS-3743 still remains unresolved. I don't see any feature in the CM to copy the edits from the live journal node to the newly added journal node. Deploying the role using CM will add the role but the Active Namenode will report that its "out of sync" with this new journal node. So before starting the new journal node, copy the edits directory from one of the existing JournalNodes to the new one. 1 - Stop the New JN.
2 - Copy all the contents under /current of Journalnode folder from a working Journalnode to the new Journal node.
3 - Add the right permission to the newly copied folders with chown and chmod.
4 - Delete the edit_inprogress file (inside the new Journal node) such that the new Journalnode will start synchronising this file with the older Journal nodes.
... View more
05-31-2022
08:18 AM
I assume you are seeing this error in the UI. Could you check the Hbase Master log to see why its down.
... View more
05-11-2022
11:40 AM
Make sure you have the truststore.jks file present under /opt/jks/truststore.jks and has the correct permissions. You can compare it with any of the working Datanode.
... View more
05-04-2022
12:26 AM
Ensure that at OS level the Namenode disk is mounted with read/write options
... View more
05-04-2022
12:12 AM
1. Check if you are able to kinit with the HDFS keytab on the Datanode host. 2. Check if the permissions are correct for the keytab and process dir. 3. If CDH, try to do a hard restart of the cloudera-scm agent. (This require all processes on the host managed by CM to be stopped) service cloudera-scm-agent hard_restart_confirmed
... View more
04-28-2022
10:04 AM
There is no direct method or property to disable tombstone marker. Major compacting the mob CF is the only way to go. There is a mob cleaner running on master that removes unreferenced mob files: The period of the cleaner chore can be configured by setting hbase.master.mob.cleaner.period to a positive integer number of seconds. It defaults to running daily. You should not need to tune it unless you have a very aggressive TTL or a very high rate of MOB updates with a correspondingly high rate of non-MOB compactions. So you can check and set the above parameter value and can manually trigger mob compaction by specifying the cf that includes mob data in hbase shell. NOTE : MOBs are mostly for writing large cells that will rarely be updated or deleted but could have TTL configured for the cell or CF.
... View more
04-28-2022
04:00 AM
Can you check if you are able to scan the hbase:namespace and hbase:acl table from the hbase shell.
... View more
- « Previous
- Next »