Stuck in upgrade control plane while upgrading ESC to 1.5.3 from 1.5.2

[#1]

Stuck in 'upgrade control plane' step while upgrading ECS to 1.5.3 from 1.5.2.

Casuses: cdp-release-post-upgrade-hook-job1 failed.

container logs are 

=========================================

**************************
**************************
https://console-cdp.apps.cdppvc1.hadoop.com
/etc/cdp/cdp_keys.json
While list environments: Something is wrong with output, Output JSON:

____ERROR__WHILE__CALLING__LIST__ENVIRONMENTS__COMMAND____ /opt/app-root/lib64/python3.9/site-packages/urllib3/connectionpool.py:1103: InsecureRequestWarning: Unverified HTTPS request is being made to host 'console-cdp.apps.cdppvc1.hadoop.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
warnings.warn(

An error occurred: 5 NOT_FOUND: No access key found with name: 1b098f81-ce2c-4342-ba02-f0e607b0d83c (Status Code: 401; Error Code: ; Service: environments; Operation: listEnvironments; Request ID: 80bb6513-3fe8-41b9-bf2f-e0a6834cabdf;)

While list environments: Something is wrong with output, Output JSON:

____ERROR__WHILE__CALLING__LIST__ENVIRONMENTS__COMMAND____ /opt/app-root/lib64/python3.9/site-packages/urllib3/connectionpool.py:1103: InsecureRequestWarning: Unverified HTTPS request is being made to host 'console-cdp.apps.cdppvc1.hadoop.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
warnings.warn(

An error occurred: 5 NOT_FOUND: No access key found with name: 1b098f81-ce2c-4342-ba02-f0e607b0d83c (Status Code: 401; Error Code: ; Service: environments; Operation: listEnvironments; Request ID: 5e7533e7-9167-40bf-870e-0880f2321829;)

While list environments: Something is wrong with output, Output JSON:

____ERROR__WHILE__CALLING__LIST__ENVIRONMENTS__COMMAND____ /opt/app-root/lib64/python3.9/site-packages/urllib3/connectionpool.py:1103: InsecureRequestWarning: Unverified HTTPS request is being made to host 'console-cdp.apps.cdppvc1.hadoop.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
warnings.warn(

An error occurred: 5 NOT_FOUND: No access key found with name: 1b098f81-ce2c-4342-ba02-f0e607b0d83c (Status Code: 401; Error Code: ; Service: environments; Operation: listEnvironments; Request ID: 029d4064-3f33-4340-bcb3-8b9d7d80f2c7;)

Failed list environments due to JSON error, Tries exhausted
Traceback (most recent call last):
File "/scripts/post_upgrade_hook.py", line 36, in <module>
data = json.loads(stdout)
File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/scripts/post_upgrade_hook.py", line 42, in <module>
raise ValueError('While list environments:\n' + str(e))
ValueError: While list environments:
Expecting value: line 1 column 1 (char 0)

===================================

Which Access key meaning in log??

What should I do to solve this problem?

[#2 · Accepted solution]

Hello @Hae 

Thank you for reaching out on this

It looks to be cdp key issue. You can confirm using the below

If the below output is returning empty results then we might need to regenerate the same and it would be better to raise a support case to further check this as this will need further debugging

>> Run the below command and look for cdp_keys.json

# kubectl --kubeconfig ocp-kubeconfig.conf -n cdp get secrets/cm.creds -o yaml

>> From the above copy the value of cdp_keys.json and pass below

# echo "cdp_key.json" | base64 --decode