Error encountered during the installation of Embedded Container Service CML: The s2i-builder pod is in a CrashLoopBackOff state

steinsgate — Tue, 21 Apr 2026 06:25:47 GMT

Hi everyone, I would like some clarification. Thank you.

Embedded Container Service 1.5.1, CML installation keeps getting stuck and eventually fails with a timeout

check the CML pod:

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml get po

We found that the status of the s2i-builder pod is CrashLoopBackOff.

check log:

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml logs s2i-builder-97d946b65-46fhc

Defaulted container "fluent-bit" out of: fluent-bit, s2i-builder, update-ca-trust (init), chowndockersock (init), chowndockercert (init) Fluent Bit v1.9.10 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2024/09/26 08:19:29] [ info] [fluent bit] version=1.9.10, commit=, pid=1 [2024/09/26 08:19:29] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128 [2024/09/26 08:19:29] [ info] [cmetrics] version=0.3.7 [2024/09/26 08:19:29] [ info] [output:stdout:stdout.0] worker #0 started [2024/09/26 08:19:29] [ info] [sp] stream processor started [2024/09/26 08:19:29] [ info] [output:forward:forward.1] worker #0 started [2024/09/26 08:19:29] [ info] [output:forward:forward.1] worker #1 started [2024/09/26 08:21:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=207397062 watch_fd=1 name=/var/fluentbit/log/s2i-builder.log [0] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286419977, {"log"=>"2024-09-26 08:21:51.465 7 INFO S2I.Builder Start getting config from env ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [1] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286428247, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Builder Finish getting config from env ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [2] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286429226, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Builder Using external registry for S2I pushing ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [3] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286430482, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Builder Start install registry creds to hosts for :0 ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [4] cml.var.fluentbit.log.s2i-builder.log: [1727338919.286431468, {"log"=>"2024-09-26 08:21:51.466 7 INFO S2I.Utils Start connecting to CDHClient GRPC endpoint ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [0] cml.var.fluentbit.log.s2i-builder.log: [1727338971.497535623, {"log"=>"2024-09-26 08:22:51.496 7 ERROR GRPC.Server gRPC connection did not become ready ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [1] cml.var.fluentbit.log.s2i-builder.log: [1727338971.497543971, {"log"=>"2024-09-26 08:22:51.497 7 ERROR S2I.Utils Finish installing s2i registry certs, Error while installing external docker pull registry certs ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [2] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500920770, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [gRPC connection did not become ready]", "namespace"=>"cml", "pod"=>"s2i-builder"}] [3] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500926289, {"log"=>"", "namespace"=>"cml", "pod"=>"s2i-builder"}] [4] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500927612, {"log"=>"goroutine 1 [running]:", "namespace"=>"cml", "pod"=>"s2i-builder"}] [5] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500928729, {"log"=>"main.runS2iBuilder()", "namespace"=>"cml", "pod"=>"s2i-builder"}] [6] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500929763, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:68 +0xd67", "namespace"=>"cml", "pod"=>"s2i-builder"}] [7] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500930716, {"log"=>"main.main()", "namespace"=>"cml", "pod"=>"s2i-builder"}] [8] cml.var.fluentbit.log.s2i-builder.log: [1727338971.500931786, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:36 +0x32", "namespace"=>"cml", "pod"=>"s2i-builder"}] [0] cml.var.fluentbit.log.s2i-builder.log: [1727338972.599122768, {"log"=>"2024-09-26 08:22:52.598 7 INFO S2I.Builder Start getting config from env ", "namespace"=>"cml", "pod"=>"s2i-builder"}]

This is an error message :

08:22:51.496 7 ERROR GRPC.Server gRPC connection did not become ready ", "namespace"=>"cml", "pod"=>"s2i-builder"}]

[1] cml.var.fluentbit.log.s2i-builder.log: [1727339032.627617491, {"log"=>"2024-09-26 08:23:52.627 7 ERROR S2I.Utils Finish installing s2i registry certs, Error while installing external docker pull registry certs ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [2] cml.var.fluentbit.log.s2i-builder.log: [1727339032.630074058, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [gRPC connection did not become ready]", "namespace"=>"cml", "pod"=>"s2i-builder"}]

After a while, connect to GRPC.

[0] cml.var.fluentbit.log.s2i-builder.log: [1727342182.219944841, {"log"=>"2024-09-26 09:16:22.218 7 INFO S2I.Utils Finish connecting to CDHClient GRPC endpoint ", "namespace"=>"cml", "pod"=>"s2i-builder"}]

Then there is this error:

[1] cml.var.fluentbit.log.s2i-builder.log: [1727342182.227297224, {"log"=>"2024-09-26 09:16:22.227 7 ERROR S2I.Utils failed to write external registry cert ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [2] cml.var.fluentbit.log.s2i-builder.log: [1727342182.227304019, {"log"=>"2024-09-26 09:16:22.227 7 ERROR S2I.Utils Finish installing s2i registry certs, Error while installing external docker pull registry certs ", "namespace"=>"cml", "pod"=>"s2i-builder"}] [3] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229502149, {"log"=>"panic: unable to install internal docker registry certificates for :0 to this host: [open /etc/docker/certs.d/ecsworker1.clover.com:5000/registry.crt: permission denied]", "namespace"=>"cml", "pod"=>"s2i-builder"}] [4] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229507778, {"log"=>"", "namespace"=>"cml", "pod"=>"s2i-builder"}] [5] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229509027, {"log"=>"goroutine 1 [running]:", "namespace"=>"cml", "pod"=>"s2i-builder"}] [6] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229510284, {"log"=>"main.runS2iBuilder()", "namespace"=>"cml", "pod"=>"s2i-builder"}] [7] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229511427, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:68 +0xd67", "namespace"=>"cml", "pod"=>"s2i-builder"}] [8] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229512764, {"log"=>"main.main()", "namespace"=>"cml", "pod"=>"s2i-builder"}] [9] cml.var.fluentbit.log.s2i-builder.log: [1727342182.229513716, {"log"=>" /root/workspace/src/github.infra.cloudera.com/Sense/cloudera-sense/services/s2i/s2i-builder/main.go:36 +0x32", "namespace"=>"cml", "pod"=>"s2i-builder"}]

We use Cloudera Default Docker Repository to install:

Re: Error encountered during the installation of Embedded Container Service CML: The s2i-builder pod is in a CrashLoopBackOff state

maaali_87 — Wed, 14 May 2025 10:04:13 GMT

I changed the permissions of the directory and it worked ,

chmod 777 /etc/docker/certs.d/node.com:5000/

question Re: Error encountered during the installation of Embedded Container Service CML: The s2i-builder pod is in a CrashLoopBackOff state in Support Questions

Error encountered during the installation of Embedded Container Service CML: The s2i-builder pod is in a CrashLoopBackOff state

Re: Error encountered during the installation of Embedded Container Service CML: The s2i-builder pod is in a CrashLoopBackOff state