CDSW 1.6.1 fails to start

[#1]

After upgrading the Cloudera Manager from 6.2.0 to 6.3.1, the CDSW won't come online anymore. In the health logs, this line stands out:

MaxRetryError: HTTPSConnectionPool(host='xxx.xxx.xxx.xxx', port=6443): Max retries exceeded with url: /api/v1/secrets (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f364478d210>: Failed to establish a new connection: [Errno 111] Connection refused',))

 The port is closed on the CDSW master node, and I couldn't find any service in the docs that depends on it. Is it Kubernetes?

Further, from cdsw validate it seems that some chains are missing from iptables:

The following chains are missing from iptables: [KUBE-EXTERNAL-SERVICES, WEAVE-NPC-EGRESS, WEAVE-NPC, WEAVE-NPC-EGRESS-ACCEPT, KUBE-SERVICES, WEAVE-NPC-INGRESS, WEAVE-NPC-EGRESS-DEFAULT, WEAVE-NPC-DEFAULT, WEAVE-NPC-EGRESS-CUSTOM]

 However, I cannot remember any step in the installation process that required to set such rules, so I assume that was automated.

Is there a way to resolve the issue without rolling back the version?

[#2 · Accepted solution]

Was able to resolve this myself while encountering more and more familiar problems. These are the important take-aways:

1. update the rest of CDH to same version as CM (6.3.1),
2. enable ipv6 for all hosts,
3. prepare the nodes using the CM,
4. restart CDSW

Probably forgot a couple of steps here, but these are the ones I remember. At least, the goal of resolving the issue without a rollback was achieved.