Member since
02-27-2023
37
Posts
3
Kudos Received
4
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 9714 | 05-09-2023 03:20 AM | |
| 5177 | 05-09-2023 03:16 AM | |
| 3833 | 03-30-2023 10:41 PM | |
| 26084 | 03-30-2023 07:25 PM |
09-04-2023
01:59 AM
Hi all, my team are currently testing on HDFS rebalancing in a HDFS cluster with two DataNode. I have read the documentation https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/scaling-namespaces/topics/hdfs-step-1-storage-group-classification.html We have a few question about how the file moving size is calculated. 1. How is cluster average utilization calculated. Do the calculation use Configured capacity or Present capacity? 2. How is individual utilization calculated. Do the calculation use configured capacity or the sum of DFS used and remaining? 3. When looking at the balancer log, it said need to move 4.98GB to make the cluster balanced. However, afterwards, the log show that the balancer decided to move 18.98GB from one host to another host. Why there is difference? Balancer log 4. We have set the DataNode balancing bandwidth to 200MiB, but as per observation, the bandwidth consumption can exceed 200MiB. Why did it happen? Please help me with my questions. Thank you in advance.
... View more
Labels:
- Labels:
-
HDFS
06-19-2023
10:36 PM
Do you confirm that you can do login from one node to another node using ssh?
... View more
05-12-2023
02:30 AM
Anyone tried that before?
... View more
05-09-2023
03:20 AM
I finally managed to solve the problem by switching the OpenJDK version from 8u362 to 8u232. However, is OpenJDK 8u232 the only version that support when working on CDP 7.1.8 with Kerberos enabled? Could someone clarify it for me please? Thank you.
... View more
05-09-2023
03:16 AM
Thank you for your help @kolli_sandeep . Finally I made an API call to interrupt the Parcels activation, then remove the corresponding Parcels file and the ./flood folder in /opt/cloudera/parcels. The parcels can successfully redistribute and activate after that.
... View more
05-09-2023
03:07 AM
Hi all, I have a CDP 7.1.8 cluster running in my environment. Recently, I would like to try enabling CM server High Availability. I studied the instruction in this link https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-ha-configure-steps.html. My question is, currently I have tried migrating the CM server to another host and I cannot see my cluster on the new CM server console until I manually restore the cluster. I really doubt that after I enabled CM server HA, when the system failover to the passive CM server, I will not see my cluster as well. Could someone clarify for me please? Thank you very much.
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Cloudera on premises
05-09-2023
02:55 AM
Hi all, I have a CDP 7.1.8 cluster running in my environment. Recently I would like to migrate the existing CM server to another host. I followed the instruction in the link https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-moving-the-cm-server-new-host2.html. After that, the CM server is successfully migrated to the target host and I can see the cluster node recognized by the new CM server. However, I cannot see the cluster appear on the new CM server console. Finally, I have to manually restore the cluster using the link https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/configuring-clusters/topics/cm-api-import-configuration.html. My question is, am I suppose to restore the cluster manually? What I expected is I can see the cluster immediately without extra work done after switching CM server. Could someone clarify for me please? Thank you.
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Cloudera on premises
05-03-2023
12:06 AM
From the system log, I found error saying unsupported Keytype. I am using OpenJDK 8u332. Do anyone know how to solve this? Thank you.
... View more
05-02-2023
03:22 AM
Hi all, I have a cluster in CDP version 7.1.8 private cloud base. The Cloudera Management service show bad health and indicate that the connection to KDC server is not available. I can regenerate missing credentials under Administration > Security > Kerberos Credentials Moreover, I can do kinit in the CM server host. Therefore I have no idea why the connection shows failure. Please let me know if I should provide further information. Thank you.
... View more
05-02-2023
03:13 AM
Hi all, I have a cluster in CDP 7.1.8 private cloud based. Today I tried migrating CM server and following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-moving-the-cm-server-new-host2.html After I manually restore the cluster following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/configuring-clusters/topics/cm-api-import-configuration.html, the cluster cannot restart. Found that in the Parcel page in CM Console, the parcels are activating but keep on loading forever. Here are the log message related to parcel from cloudera-scm-agent.log: [02/May/2023 03:29:55 -0400] 6258 MainThread parcel ERROR Error while attempting to modify permissions of file '/opt/ cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'.
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel.py", line 586, in ensure_permissions
file = cmf.util.validate_and_open_fd(path, self.get_parcel_home(parcel))
OSError: [Errno 2] No such file or directory: '/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hadoop-0.20-mapreduc e/sbin/Linux/task-controller'
[02/May/2023 03:29:55 -0400] 6258 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[02/May/2023 03:29:55 -0400] 6258 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[02/May/2023 03:29:55 -0400] 6258 MainThread throttling_logger WARNING Failed parsing alternatives line: sqoop-export string index out of range link currently points to /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/bin/sqoop-export
[02/May/2023 03:29:59 -0400] 6258 MainThread parcel_cache INFO Deleting unmanaged parcel CDH-7.1.8-1.cdh7.1.8.p0.30990532
[02/May/2023 03:30:40 -0400] 6258 MainThread parcel_cache INFO Deleting unmanaged parcel SPARK3-3.3.0.3.3.7180.0-274-1.p0. 31212967
[02/May/2023 03:30:40 -0400] 6258 MainThread parcel INFO prepare_environment begin: {}, [], []
[02/May/2023 03:30:40 -0400] 6258 MainThread parcel INFO No parcels activated for use
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:44:43 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:43 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 04:57:36 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {}, [], []
[02/May/2023 04:57:36 -0400] 6258 __run_queue parcel INFO No parcels activated for use
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:05:21 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:21 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0- 274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel INFO Service does not request any parcels
[02/May/2023 05:30:15 -0400] 28744 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/c onfig.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[02/May/2023 05:30:19 -0400] 28744 MainThread agent INFO Previously active parcels: {}
[02/May/2023 05:30:19 -0400] 28744 MainThread agent INFO Using parcels directory from server provided value: /opt/c loudera/parcels
[02/May/2023 05:30:19 -0400] 28744 MainThread parcel INFO Agent does create users/groups
[02/May/2023 05:30:19 -0400] 28744 MainThread parcel INFO Agent does parcel permissions
[02/May/2023 05:30:19 -0400] 28744 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[02/May/2023 05:30:19 -0400] 28744 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[02/May/2023 05:30:19 -0400] 28744 MainThread throttling_logger WARNING Failed parsing alternatives line: sqoop-export string index out of range link currently points to /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/bin/sqoop-export
[02/May/2023 05:31:59 -0400] 29223 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/c onfig.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[02/May/2023 05:32:03 -0400] 29223 MainThread agent INFO Previously active parcels: {}
[02/May/2023 05:32:03 -0400] 29223 MainThread agent INFO Using parcels directory from server provided value: /opt/c loudera/parcels Please let me know if I need to provide any further information. Thank you.
... View more
Labels:
- Labels:
-
Cloudera
-
Cloudera Manager