Error encountered during the installation of Embedded Container Service CML: The s2i-builder pod is in a CrashLoopBackOff state

[#1]

Hi everyone, I would like some clarification. Thank you.

Embedded Container Service 1.5.1, CML installation keeps getting stuck and eventually fails with a timeout

check the CML pod:

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml get po

We found that the status of the s2i-builder pod is CrashLoopBackOff.

check log:

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml logs s2i-builder-97d946b65-46fhc

Defaulted container "fluent-bit" out of: fluent-bit, s2i-builder, update-ca-trust (init), chowndockersock (init), chowndockercert (init)
Fluent Bit v1.9.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2024/09/26 08:19:29] [ info] [fluent bit] version=1.9.10, commit=, pid=1
[2024/09/26 08:19:29] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2024/09/26 08:19:29] [ info] [cmetrics] version=0.3.7
[2024/09/26 08:19:29] [ info] [output:stdout:stdout.0] worker #0 started
[2024/09/26 08:19:29] [ info] [sp] stream processor started
[2024/09/26 08:19:29] [ info] [output:forward:forward.1] worker #0 started
[2024/09/26 08:19:29] [ info] [output:forward:forward.1] worker #1 started
[2024/09/26 08:21:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=207397062 watch_fd=1 name=/var/fluentbit/log/s2i-builder.log
[0] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286419977, {"log"=>"2024-09-26 08:21:51.465	7	INFO   	S2I.Builder                   	Start  getting config from env	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[1] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286428247, {"log"=>"2024-09-26 08:21:51.466	7	INFO   	S2I.Builder                   	Finish getting config from env	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286429226, {"log"=>"2024-09-26 08:21:51.466	7	INFO   	S2I.Builder                   	Using external registry for S2I pushing	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[3] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286430482, {"log"=>"2024-09-26 08:21:51.466	7	INFO   	S2I.Builder                   	Start  install registry creds to hosts for :0	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[4] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286431468, {"log"=>"2024-09-26 08:21:51.466	7	INFO   	S2I.Utils                     	Start  connecting to CDHClient GRPC endpoint	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[0] cml.var.fluentbit.log.s2i-builder.log: [1727338971.497535623, {"log"=>"2024-09-26 08:22:51.496	7	ERROR  	GRPC.Server                   	gRPC connection did not become ready	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[1] cml.var.fluentbit.log.s2i-builder.log: [1727338971.497543971, {"log"=>"2024-09-26 08:22:51.497	7	ERROR  	S2I.Utils                     	Finish installing s2i registry certs, Error while installing external docker pull registry certs 	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500920770, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [gRPC connection did not become ready]", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[3] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500926289, {"log"=>"", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[4] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500927612, {"log"=>"goroutine 1 [running]:", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[5] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500928729, {"log"=>"main.runS2iBuilder()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[6] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500929763, {"log"=>"	/root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:68 +0xd67", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[7] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500930716, {"log"=>"main.main()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[8] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500931786, {"log"=>"	/root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:36 +0x32", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[0] cml.var.fluentbit.log.s2i-builder.log: [1727338972.599122768, {"log"=>"2024-09-26 08:22:52.598	7	INFO   	S2I.Builder                   	Start  getting config from env	", "namespace"=>"cml", "pod"=>"s2i-builder"}]

This is an error message : 

08:22:51.496	7	ERROR  	GRPC.Server                   	gRPC connection did not become ready	", "namespace"=>"cml", "pod"=>"s2i-builder"}]

[1] cml.var.fluentbit.log.s2i-builder.log: [1727339032.627617491, {"log"=>"2024-09-26 08:23:52.627	7	ERROR  	S2I.Utils                     	Finish installing s2i registry certs, Error while installing external docker pull registry certs 	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727339032.630074058, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [gRPC connection did not become ready]", "namespace"=>"cml", "pod"=>"s2i-builder"}]

After a while, connect to GRPC.

[0] cml.var.fluentbit.log.s2i-builder.log: [1727342182.219944841, {"log"=>"2024-09-26 09:16:22.218	7	INFO   	S2I.Utils                     	Finish connecting to CDHClient GRPC endpoint	", "namespace"=>"cml", "pod"=>"s2i-builder"}]

Then there is this error:

[1] cml.var.fluentbit.log.s2i-builder.log: [1727342182.227297224, {"log"=>"2024-09-26 09:16:22.227	7	ERROR  	S2I.Utils                     	failed to write external registry cert	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[2] cml.var.fluentbit.log.s2i-builder.log: [1727342182.227304019, {"log"=>"2024-09-26 09:16:22.227	7	ERROR  	S2I.Utils                     	Finish installing s2i registry certs, Error while installing external docker pull registry certs 	", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[3] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229502149, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [open /etc/docker/certs.d/ecsworker1.clover.com:5000/registry.crt: permission denied]", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[4] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229507778, {"log"=>"", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[5] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229509027, {"log"=>"goroutine 1 [running]:", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[6] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229510284, {"log"=>"main.runS2iBuilder()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[7] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229511427, {"log"=>"	/root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:68 +0xd67", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[8] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229512764, {"log"=>"main.main()", "namespace"=>"cml", "pod"=>"s2i-builder"}]
[9] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229513716, {"log"=>"	/root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:36 +0x32", "namespace"=>"cml", "pod"=>"s2i-builder"}]

We use Cloudera Default Docker Repository to install:

[#2]

I changed the permissions of the directory and it worked ,

chmod 777 /etc/docker/certs.d/node.com:5000/