"No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

[#1]

Hi All,

When Kerberos and TLS is enabled on the CDH cluster, role logs are not reported in Cloudera Manager under the respective services. CM reports 'No log messages at the specified URL' for each of the following:

1. Role Log File
2. Stdout
3. Stderr

Interestingly, the log files mentioned on the log pages (on CM) exist on the underlined host and contain all & correct log entries; They just don't get reported at CM service log pages. When tried downloading full log files, it throws error:

HTTP ERROR 403

Problem accessing /cmf/process/335/logs. Reason:

    Unexpected end of file from server
The server declined access to the page or resource.

Powered by Jetty:// 9.4.14.v20181114

Same error is observed when attempted fetching logs using CM API.

Environment:

CDH	6.2
OS	Redhat 7.7

[#2 · Accepted solution]

@SandeepSingh This looks like the issue with TLS. 

Eventhough the flag 'Use TLS Authentication of Agents to Server' in CM WebUI is not set, the following flag must be set for status_server to use TLS protocol using port 9000. Go to the /opt/cloudera/security/x509/ directory and use 'pem' and 'key' file under that directory. You may also have to use the password file for the private key if there is one.

Then edit the /etc/cloudera-scm-agent/config.ini file with below parameters.

# PEM file containing client private key.
client_key_file=

# If client_keypw_cmd isn't specified, instead a text file containing the client private key password can be used.
client_keypw_file=

# PEM file containing client certificate.
client_cert_file=/etc/cdep-ssl-conf/CA_STANDARD/cm_server-cert.pem

verify_cert_file=

Restart of the status_server is required

cd /var/run/cloudera-scm-agent/supervisord
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

In addition, restart of the cloudera-scm-agent is also needed
service cloudera-scm-agent restart

[#3 · Accepted solution]

@SandeepSingh This looks like the issue with TLS. 

Eventhough the flag 'Use TLS Authentication of Agents to Server' in CM WebUI is not set, the following flag must be set for status_server to use TLS protocol using port 9000. Go to the /opt/cloudera/security/x509/ directory and use 'pem' and 'key' file under that directory. You may also have to use the password file for the private key if there is one.

Then edit the /etc/cloudera-scm-agent/config.ini file with below parameters.

# PEM file containing client private key.
client_key_file=

# If client_keypw_cmd isn't specified, instead a text file containing the client private key password can be used.
client_keypw_file=

# PEM file containing client certificate.
client_cert_file=/etc/cdep-ssl-conf/CA_STANDARD/cm_server-cert.pem

verify_cert_file=

Restart of the status_server is required

cd /var/run/cloudera-scm-agent/supervisord
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

In addition, restart of the cloudera-scm-agent is also needed
service cloudera-scm-agent restart

[#4 · Accepted solution]

@GangWar Thanks for your suggestion.

All the parameters except the following one were already set in /etc/cloudera-scm-agent/config.ini

verify_cert_file

Apparently, the only reason why agent wasn't serving requests for logs was because the above flat wasn't set.

The moment we configured the flag verify_cert_file and restarted agent, it started serving logs correctly.