Member since
09-12-2023
21
Posts
16
Kudos Received
0
Solutions
10-01-2024
03:04 AM
1 Kudo
Hello everyone, We would like to try disabling Hive Kerberos on CDP 7.1.7. Here are the parameters we tried disabling: Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml hive.server2.authentication.kerberos.principal hive.server2.authentication.kerberos.keytab hive.server2.authentication NONE hive.security.authorization.enabled false hive.server2.enable.impersonation false hive.metastore.sasl.enabled false Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml hadoop.security.authentication simple hadoop.security.authorization false After turning off the following parameters and restarting HiveServer2, we noticed that HiveServer2 still seems to fail due to Kerberos authentication. Could you please advise on how to completely disable Kerberos for HiveServer2? Thank you.
... View more
09-26-2024
03:30 AM
1 Kudo
Hi everyone, I would like some clarification. Thank you. Embedded Container Service 1.5.1, CML installation keeps getting stuck and eventually fails with a timeout check the CML pod: /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml get po We found that the status of the s2i-builder pod is CrashLoopBackOff. check log: /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml logs s2i-builder-97d946b65-46fhc Defaulted container "fluent-bit" out of: fluent-bit, s2i-builder, update-ca-trust (init), chowndockersock (init), chowndockercert (init)
Fluent Bit v1.9.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2024/09/26 08:19:29] [ info] [fluent bit] version=1.9.10, commit=, pid=1
[2024/09/26 08:19:29] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2024/09/26 08:19:29] [ info] [cmetrics] version=0.3.7
[2024/09/26 08:19:29] [ info] [output:stdout:stdout.0] worker #0 started
[2024/09/26 08:19:29] [ info] [sp] stream processor started
[2024/09/26 08:19:29] [ info] [output:forward:forward.1] worker #0 started
[2024/09/26 08:19:29] [ info] [output:forward:forward.1] worker #1 started
[2024/09/26 08:21:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=207397062 watch_fd=1 name=/var/fluentbit/log/s2i-builder.log
[0] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286419977, {"log"=>"2024-09-26 08:21:51.465 7 INFO S2I.Builder Start getting config from env ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[1] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286428247, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Builder Finish getting config from env ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286429226, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Builder Using external registry for S2I pushing ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[3] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286430482, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Builder Start install registry creds to hosts for :0 ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[4] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286431468, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Utils Start connecting to CDHClient GRPC endpoint ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[0] cml.var.fluentbit.log.s2i-builder.log: [1727338971.497535623, {"log"=>"2024-09-26 08:22:51.496 7 ERROR GRPC.Server gRPC connection did not become ready ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[1] cml.var.fluentbit.log.s2i-builder.log: [1727338971.497543971, {"log"=>"2024-09-26 08:22:51.497 7 ERROR S2I.Utils Finish installing s2i registry certs, Error while installing external docker pull registry certs ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500920770, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [gRPC connection did not become ready]", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[3] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500926289, {"log"=>"", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[4] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500927612, {"log"=>"goroutine 1 [running]:", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[5] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500928729, {"log"=>"main.runS2iBuilder()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[6] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500929763, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:68 +0xd67", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[7] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500930716, {"log"=>"main.main()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[8] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500931786, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:36 +0x32", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[0] cml.var.fluentbit.log.s2i-builder.log: [1727338972.599122768, {"log"=>"2024-09-26 08:22:52.598 7 INFO S2I.Builder Start getting config from env ", "namespace"=>"cml", "pod"=>"s2i-builder"}] This is an error message : 08:22:51.496 7 ERROR GRPC.Server gRPC connection did not become ready ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [1] cml.var.fluentbit.log.s2i-builder.log: [1727339032.627617491, {"log"=>"2024-09-26 08:23:52.627 7 ERROR S2I.Utils Finish installing s2i registry certs, Error while installing external docker pull registry certs ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727339032.630074058, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [gRPC connection did not become ready]", "namespace"=>"cml", "pod"=>"s2i-builder"}] After a while, connect to GRPC. [0] cml.var.fluentbit.log.s2i-builder.log: [1727342182.219944841, {"log"=>"2024-09-26 09:16:22.218 7 INFO S2I.Utils Finish connecting to CDHClient GRPC endpoint ", "namespace"=>"cml", "pod"=>"s2i-builder"}] Then there is this error: [1] cml.var.fluentbit.log.s2i-builder.log: [1727342182.227297224, {"log"=>"2024-09-26 09:16:22.227 7 ERROR S2I.Utils failed to write external registry cert ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727342182.227304019, {"log"=>"2024-09-26 09:16:22.227 7 ERROR S2I.Utils Finish installing s2i registry certs, Error while installing external docker pull registry certs ", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[3] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229502149, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [open /etc/docker/certs.d/ecsworker1.clover.com:5000/registry.crt: permission denied]", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[4] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229507778, {"log"=>"", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[5] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229509027, {"log"=>"goroutine 1 [running]:", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[6] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229510284, {"log"=>"main.runS2iBuilder()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[7] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229511427, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:68 +0xd67", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[8] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229512764, {"log"=>"main.main()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[9] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229513716, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:36 +0x32", "namespace"=>"cml", "pod"=>"s2i-builder"}] We use Cloudera Default Docker Repository to install:
... View more
05-02-2024
11:52 PM
1 Kudo
Hi everyone, I would like some clarification. Thank you. When we enabled Kerberos, we encountered some errors during the "start clusters" phase in the command details, specifically when starting HBase. Subsequently, we examined the HBase log: cat /var/log/hbase/hbase-cmf-hbase-REGIONSERVER-cdp717w2.am.ocp.poc.log.out 2024-05-02 23:14:41,118 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: regionserver:16020-0x30046d0bb0d003e, quorum=cdp717m1.am.ocp.poc:2181,cdp717mo.am.ocp.poc:2181,cdp717w0.am.ocp.poc:2181, baseZNode=/hbase Unable to get data of znode /hbase/running
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running
at org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:358)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:659)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:635)
at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.start(ZKNodeTracker.java:79)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:659)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:3066)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3084)
2024-05-02 23:14:41,122 ERROR org.apache.hadoop.hbase.zookeeper.ZKWatcher: regionserver:16020-0x30046d0bb0d003e, quorum=cdp717m1.am.ocp.poc:2181,cdp717mo.am.ocp.poc:2181,cdp717w0.am.ocp.poc:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running When we go to zookeeper cmd check the path: /hbase/running We found that there was a permission issue so we couldn’t get it. We then speculated that we might need to disable the Zookeeper's ACL and restart Zookeeper. After resuming using the wizard, we were able to start HBase and complete enabling Kerberos for the cluster. Then because of the skip acl, we have permissions to /hbase/running I would like some clarification when encountering this error, is it advisable to skip the ACL directly? Or are there any other methods that would be more appropriate? Regards
... View more
04-23-2024
01:02 AM
1 Kudo
Hello @upadhyayk04 I see, thank you.
... View more
04-10-2024
12:01 AM
1 Kudo
Hi everyone, I would like some clarification. Thank you. CDP recommends disabling the 'tuned' service for performance optimization. Disable the tuned Service The suggestion to disable 'tuned' stems from its capability to transition the CPU into c-state idle mode, potentially leading to decreased performance. Disable the tuned Service on RHEL 8 We would like some clarification about the explanation regarding 'tuned', such as its potential impact on Hadoop. Thank you. For example, THP: Disabling Transparent Hugepages (THP) Most Linux platforms supported by Cloudera Runtime include a feature called transparent hugepages, which interacts poorly with Hadoop workloads and can seriously degrade performance. For example, swappiness: Setting the vm.swappiness Linux Kernel Parameter On most systems, vm.swappiness is set to 60 by default. This is not suitable for Hadoop clusters because processes are sometimes swapped even when enough memory is available. Regards
... View more
04-08-2024
09:18 PM
2 Kudos
Hello @upadhyayk04 Thank you, marked as resolved.
... View more
04-08-2024
01:06 AM
1 Kudo
Hello @upadhyayk04 Thank you for your reply. We are using CM 7.11.3, CDP 7.1.9, ECS 1.5.2. After reinstalling and deleting the rancher folder, we have successfully passed through this error. But we don’t know the actual reason. Currently, we are still investigating where the DNS error might be occurring.
... View more
04-03-2024
11:27 AM
Hello @upadhyayk04 I would like some clarification. Thank you. I found this warning, I don’t know if it’s the reason.
... View more
04-03-2024
03:49 AM
1 Kudo
Hello @upadhyayk04 Thank you very much for your response. After reinstalling, we encountered this issue. https://community.cloudera.com/t5/Support-Questions/Failure-to-Install-Embedded-Container-Service-ECS/td-p/385954 Could you please take a look at it when you have some free time? Thank you.
... View more
04-03-2024
03:44 AM
1 Kudo
Hi everyone, I would like some clarification. Thank you. We are encountering a blockage during the installation of the Embedded Container Service (ECS). Encountering the following error: 2024/04/03 17:29:48 kubectl get secret cm.creds -n cdp
2024/04/03 17:39:38 Error while getting account id. Get "https://console-cdp.apps.ecsm0.am.ocp.poc/authenticate/login": dial tcp: lookup console-cdp.apps.ecsm0.am.ocp.poc on 192.168.30.206:53: no such host
2024/04/03 17:39:38 Retry GetAccountId after 10 seconds...
2024/04/03 17:39:48 There is an error while retrieving the account id.
panic: There is an error while retrieving the account id.
goroutine 1 [running]:
log.Panicf({0x176705e?, 0xc00074c4c0?}, {0x0?, 0xc00074c4c0?, 0xc00074c4ea?})
/grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/thirdparty/go/src/log/log.go:391 +0x67
main.GetAccountId({0xc00074c4c0, 0x3c})
/grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/src/go/create-cm-cred.go:129 +0x1ef
main.(*CdpInstaller).CreateCmCred(0xc000605e28?)
/grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/src/go/create-cm-cred.go:51 +0x14a
main.(*CdpInstaller).installControlPlane(0xc000605e28)
/grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/src/go/helm-install-all.go:570 +0x36b
main.main()
/grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/src/go/main.go:26 +0xde
2024/04/03 17:39:48 =========================================================================================
2024/04/03 17:39:48 Report workflow status:
2024/04/03 17:39:48 {
"WorkflowStatusArray": null,
"AllFlowsSucceeded": false Do we need to configure anything for DNS for Embedded Container Service (ECS)? Or do we need to configure anything in k8s pods?
... View more
Labels: