Resolution of Failed Knox Gateway Start During CDP Upgrade

[#1]

 Summary

This article details the resolution of a startup failure for Knox Gateway roles during a Data Hub upgrade, involving the reconfiguration of a missing knox_pac4j_password.

Symptoms

• While upgrading the Data Hub cluster, Knox Gateway services failed to start.
• Error message: java.lang.IllegalArgumentException: The variable [${knox_pac4j_password}] does not have a corresponding value.
• The upgrade process was halted due to this failure.

Cause

• The probable cause of the startup failure was an improperly generated or missing knox_pac4j_password during the upgrade process.
• The built-in upgrade handler should have automatically generated the Knox secret, but it did not.

Instructions

To resolve the issue, the following steps were taken:

1. Verify if an entry for knox_pac4j_password exists in the Cloudera Manager database.
   • Execute SQL query: SELECT * FROM configs WHERE attr LIKE '%knox_pac4j_password%';
2. Generate a 16-bit UUID random password.
   • Use OpenSSL: openssl rand -base64 16
3. Create a JSON file (test.json) with the generated password as follows:
   { "items": [ { "name": "knox_pac4j_password", "value": "[Generated_Password]" } ] }
4. Confirm the Knox service name via Cloudera API call.
   • Use curl to fetch service details.
5. Update the Cloudera Manager configuration with the new password using Cloudera API.
   • Use curl with the PUT method to update the configuration.
6. Confirm the new value is persisted in the Cloudera Manager database.
   • Execute SQL query: SELECT * FROM configs WHERE attr LIKE '%knox_pac4j_password%';
7. Restart or start the Knox service.
   • Use Cloudera Manager UI or API to restart the service.

Following these steps successfully resolved the issue, allowing the Knox service to start without errors, and the upgrade process was able to continue.