Summary This article details the resolution of a startup failure for Knox Gateway roles during a Data Hub upgrade, involving the reconfiguration of a missing knox_pac4j_password. Symptoms While upgrading the Data Hub cluster, Knox Gateway services failed to start. Error message: java.lang.IllegalArgumentException: The variable [${knox_pac4j_password}] does not have a corresponding value. The upgrade process was halted due to this failure. Cause The probable cause of the startup failure was an improperly generated or missing knox_pac4j_password during the upgrade process. The built-in upgrade handler should have automatically generated the Knox secret, but it did not. Instructions To resolve the issue, the following steps were taken: Verify if an entry for knox_pac4j_password exists in the Cloudera Manager database. Execute SQL query: SELECT * FROM configs WHERE attr LIKE '%knox_pac4j_password%'; Generate a 16-bit UUID random password. Use OpenSSL: openssl rand -base64 16 Create a JSON file (test.json) with the generated password as follows: { "items": [ { "name": "knox_pac4j_password", "value": "[Generated_Password]" } ] }   Confirm the Knox service name via Cloudera API call. Use curl to fetch service details. Update the Cloudera Manager configuration with the new password using Cloudera API. Use curl with the PUT method to update the configuration. Confirm the new value is persisted in the Cloudera Manager database. Execute SQL query: SELECT * FROM configs WHERE attr LIKE '%knox_pac4j_password%'; Restart or start the Knox service. Use Cloudera Manager UI or API to restart the service. Following these steps successfully resolved the issue, allowing the Knox service to start without errors, and the upgrade process was able to continue. ... View more

Summary Resolved service errors on Datanode#09 by identifying and addressing NFS mount issues. The Cloudera agent was reconfigured to bypass NFS checks temporarily, allowing services to return to a healthy state. Symptoms Multiple service errors reported on Datanode#09. Cloudera agent was not in contact with Cloudera Manager. Filesystem usage for certain nodev filesystems was unknown due to an inactive worker process. Cause The NFS partition was not properly mounted on the host, which led to problems with the Cloudera agent's health checks. The Cloudera agent configuration was set to monitor NFS mounts (monitored_nodev_filesystem_types=nfs,nfs4,tmpfs), which failed due to the unmounted NFS partition. Instructions Temporarily comment out the NFS monitoring line in the Cloudera agent configuration file to bypass the check and restore agent communication: Open /etc/cloudera-scm-agent/config.ini using a text editor. Comment out the line monitored_nodev_filesystem_types=nfs,nfs4,tmpfs. Restart the Cloudera agent service: service cloudera-scm-agent restart. Once the NFS mount point is recovered with the help of the OS team, restore the original configuration: Uncomment the previously commented line in /etc/cloudera-scm-agent/config.ini. Restart the Cloudera agent service to apply changes. ... View more