Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

"No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

avatar

Hi All,

 

When Kerberos and TLS is enabled on the CDH cluster, role logs are not reported in Cloudera Manager under the respective services. CM reports 'No log messages at the specified URL' for each of the following:

  1. Role Log File
  2. Stdout
  3. Stderr

Interestingly, the log files mentioned on the log pages (on CM) exist on the underlined host and contain all & correct log entries; They just don't get reported at CM service log pages. When tried downloading full log files, it throws error:

HTTP ERROR 403

Problem accessing /cmf/process/335/logs. Reason:

    Unexpected end of file from server
The server declined access to the page or resource.

 

Powered by Jetty:// 9.4.14.v20181114

 

Same error is observed when attempted fetching logs using CM API.

 

Environment:

CDH 6.2
OS Redhat 7.7

 

2 ACCEPTED SOLUTIONS

avatar
Master Guru

@SandeepSingh This looks like the issue with TLS. 

Eventhough the flag 'Use TLS Authentication of Agents to Server' in CM WebUI is not set, the following flag must be set for status_server to use TLS protocol using port 9000. Go to the /opt/cloudera/security/x509/ directory and use 'pem' and 'key' file under that directory. You may also have to use the password file for the private key if there is one.

Then edit the /etc/cloudera-scm-agent/config.ini file with below parameters.

# PEM file containing client private key.
client_key_file=

# If client_keypw_cmd isn't specified, instead a text file containing the client private key password can be used.
client_keypw_file=

# PEM file containing client certificate.
client_cert_file=/etc/cdep-ssl-conf/CA_STANDARD/cm_server-cert.pem

verify_cert_file=

Restart of the status_server is required

cd /var/run/cloudera-scm-agent/supervisord
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

In addition, restart of the cloudera-scm-agent is also needed
service cloudera-scm-agent restart

Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

avatar

@GangWar Thanks for your suggestion.

All the parameters except the following one were already set in /etc/cloudera-scm-agent/config.ini

verify_cert_file

Apparently, the only reason why agent wasn't serving requests for logs was because the above flat wasn't set.

The moment we configured the flag verify_cert_file and restarted agent, it started serving logs correctly.

View solution in original post

11 REPLIES 11

avatar
Master Guru

@SandeepSingh This looks like the issue with TLS. 

Eventhough the flag 'Use TLS Authentication of Agents to Server' in CM WebUI is not set, the following flag must be set for status_server to use TLS protocol using port 9000. Go to the /opt/cloudera/security/x509/ directory and use 'pem' and 'key' file under that directory. You may also have to use the password file for the private key if there is one.

Then edit the /etc/cloudera-scm-agent/config.ini file with below parameters.

# PEM file containing client private key.
client_key_file=

# If client_keypw_cmd isn't specified, instead a text file containing the client private key password can be used.
client_keypw_file=

# PEM file containing client certificate.
client_cert_file=/etc/cdep-ssl-conf/CA_STANDARD/cm_server-cert.pem

verify_cert_file=

Restart of the status_server is required

cd /var/run/cloudera-scm-agent/supervisord
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

In addition, restart of the cloudera-scm-agent is also needed
service cloudera-scm-agent restart

Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar

@GangWar Thanks for your suggestion.

All the parameters except the following one were already set in /etc/cloudera-scm-agent/config.ini

verify_cert_file

Apparently, the only reason why agent wasn't serving requests for logs was because the above flat wasn't set.

The moment we configured the flag verify_cert_file and restarted agent, it started serving logs correctly.