Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

"No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

avatar

Hi All,

 

When Kerberos and TLS is enabled on the CDH cluster, role logs are not reported in Cloudera Manager under the respective services. CM reports 'No log messages at the specified URL' for each of the following:

  1. Role Log File
  2. Stdout
  3. Stderr

Interestingly, the log files mentioned on the log pages (on CM) exist on the underlined host and contain all & correct log entries; They just don't get reported at CM service log pages. When tried downloading full log files, it throws error:

HTTP ERROR 403

Problem accessing /cmf/process/335/logs. Reason:

    Unexpected end of file from server
The server declined access to the page or resource.

 

Powered by Jetty:// 9.4.14.v20181114

 

Same error is observed when attempted fetching logs using CM API.

 

Environment:

CDH 6.2
OS Redhat 7.7

 

2 ACCEPTED SOLUTIONS

avatar
Master Guru

@SandeepSingh This looks like the issue with TLS. 

Eventhough the flag 'Use TLS Authentication of Agents to Server' in CM WebUI is not set, the following flag must be set for status_server to use TLS protocol using port 9000. Go to the /opt/cloudera/security/x509/ directory and use 'pem' and 'key' file under that directory. You may also have to use the password file for the private key if there is one.

Then edit the /etc/cloudera-scm-agent/config.ini file with below parameters.

# PEM file containing client private key.
client_key_file=

# If client_keypw_cmd isn't specified, instead a text file containing the client private key password can be used.
client_keypw_file=

# PEM file containing client certificate.
client_cert_file=/etc/cdep-ssl-conf/CA_STANDARD/cm_server-cert.pem

verify_cert_file=

Restart of the status_server is required

cd /var/run/cloudera-scm-agent/supervisord
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

In addition, restart of the cloudera-scm-agent is also needed
service cloudera-scm-agent restart

Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

avatar

@GangWar Thanks for your suggestion.

All the parameters except the following one were already set in /etc/cloudera-scm-agent/config.ini

verify_cert_file

Apparently, the only reason why agent wasn't serving requests for logs was because the above flat wasn't set.

The moment we configured the flag verify_cert_file and restarted agent, it started serving logs correctly.

View solution in original post

11 REPLIES 11

avatar
Rising Star

Hi Sandeep,

Can you please check the owner and permissions of "/var/run/cloudera-scm-server" properties whether it is  root or cloudera-scm ??

Thanks and Regards,
Bhuvan.

avatar

Hi @Bhuv

 

Here are the owner/permission details:

 

drwxr-xr-x.  2 cloudera-scm cloudera-scm   40 Sep 24 12:05 cloudera-scm-server

 

Regards,

Sandeep

avatar
Master Guru

@SandeepSingh  If the same error is also can be observed from the API then this is most probably the issue with TLS. However could you please try to narrow down the issue by following methods:

  • Run the following command on every Agent host with supervisorctl: 

 

/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server​

 

Please let us know how this goes. If the issue remains then please run the Host Inspector and share the result.


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Guru

Hi @SandeepSingh ,

 

Could you please share your Cloudera Manager version? I see you mentioned CDH version is 6.2. What about CM version? There was a bug in CM 6.1 where under certain condition the CM Server fails to contact agent when TLS is on. I am wondering if you are hitting that bug.

 

Thanks,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

avatar
Hi @lwang

CM version in this case is 6.2, so I suspect if I am hitting the bug you've mentioned; Unless this wasn't fixed in 6.2 too. Do you have an info?

Regards
Sandeep

avatar
Master Guru

@SandeepSingh  Have you tried to run below command in problematic host as stated earlier?

/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server​

This bug has been fixed in the 6.2. I may be possible the status server did not being restarted after making the changes. Please run the command and also follow the other steps stated in previous post and let us know how this goes. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar

Hi @GangWar

 

I tried the both of the following command; however result was exactly the same.

 

/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server​

 

I have indeed Enable Kerberos Authentication for HTTP Web-Consoles option checked-in and I already tried configuring SPNEGO in the browsers (Mozilla Firefox as well Chrome) as stated in Cloudera documentation; no luck.

 

Host Inspector also failed to complete on all hosts of the cluster. Error Details:

IOException thrown while collecting data from host: Read timed out
stdout
"etcKrbConfMessages" : [ ], "extantInitdErrors" : [ ], "groupData" : "hdfs:x:11004:hive,impala\nmapred:x:11014:\nzookeeper:x:11022:\noozie:x:11015:\nhbase:x:11003:\nhue:x:11007:\ncloudera-scm:x:991:\nhadoop:x:11002:hdfs,hive,impala,mapred,yarn\nhive:x:11005:impala\nsqoop:x:11019:\nimpala:x:11008:\nyarn:x:11021:\nhttpfs:x:11006:\nsentry:x:11018:\nsolr:x:11016:\nspark:x:11017:\nkms:x:11010:\n", "hostDnsErrors" : [ ], "hostname" : "dwz-06.nonprod.lan", "jceStrength" : 0, "kdcConnectionMessages" : [ ], "kernelVersion" : "3.10.0-1062.1.2.el7.x86_64", "kernelVersionException" : null, "localHostIpError" : null, "localhostIp" : "127.0.0.1", "nowMillis" : 1571833401403, "psycopg2Version" : "2.7.5", "psycopg2VersionException" : null, "psycopg2VersionOk" : true, "pythonVersionException" : null, "pythonVersionOk" : true, "pythonVersionString" : "2.7", "rhelRelease" : "Red Hat Enterprise Linux Server release 7.7 (Maipo)", "runExceptions" : [ ], "swappiness" : "60", "swappinessException" : null, "timeZone" : "UTC+01:00", "transparentHugePagesDefrag" : "[always] madvise never", "transparentHugePagesEnabled" : "always madvise [never]", "transparentHugePagesException" : null, "transparentHugePagesPath" : "/sys/kernel/mm/transparent_hugepage", "userData" : "hdfs:x:980:11004::/home/hdfs:/bin/bash\nmapred:x:977:11014::/home/mapred:/bin/bash\nzookeeper:x:981:11022::/home/zookeeper:/bin/bash\noozie:x:986:11015::/home/oozie:/bin/bash\nhbase:x:994:11003::/home/hbase:/bin/bash\nhue:x:992:11007::/home/hue:/bin/bash\ncloudera-scm:x:974:991:Cloudera Manager:/var/lib/cloudera-scm-server:/sbin/nologin\nsqoop:x:982:11019::/home/sqoop:/bin/bash\nsqoop2:x:976:11019::/home/sqoop2:/bin/bash\nimpala:x:978:11008::/home/impala:/bin/bash\nyarn:x:975:11021::/home/yarn:/bin/bash\nhttpfs:x:993:11006::/home/httpfs:/bin/bash\nsentry:x:983:11018::/home/sentry:/bin/bash\nsolr:x:985:11016::/home/solr:/bin/bash\nspark:x:984:11017::/home/spark:/bin/bash\nkms:x:990:11010::/home/kms:/bin/bash\n" }
stderr
+ [[ inspector == \f\i\r\e\h\o\s\e ]] + [[ inspector == \e\v\e\n\t\s\e\r\v\e\r ]] + [[ inspector == \a\l\e\r\t\p\u\b\l\i\s\h\e\r ]] + [[ inspector == \h\e\a\d\l\a\m\p ]] + [[ inspector == \t\e\l\e\m\e\t\r\y\p\u\b\l\i\s\h\e\r ]] + [[ inspector == \t\e\s\t\-\d\b\u\s\-\c\o\n\n\e\c\t\i\o\n ]] + [[ inspector == \i\n\s\p\e\c\t\o\r ]] + shift ++ pwd + MGMT_CLASSPATH='/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' + echo_and_exec /usr/java/jdk1.8.0_131/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT + echo 'Executing: /usr/java/jdk1.8.0_131/bin/java' -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT + exec /usr/java/jdk1.8.0_131/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT

 

Here is what CM shows on Host Inspection result page:

 

Validations

 Inspector failed on the following hosts...
View Details
  • dwz-[01-07].nonprod.lan: IOException thrown while collecting data from host: Read timed out
Inspector ran on 0 hosts.
 The inspector failed to run on all hosts.
 0 hosts are running CDH 5 and 7 hosts are running CDH 6.
 All checked hosts in each cluster are running the same version of components.
 All managed hosts have consistent versions of Java.
 All checked Cloudera Management Daemons versions are consistent with the server.
 All checked Cloudera Management Agents versions are consistent with the server.

 

Version Summary

AfisPOCDataLake

All HostsComponent Version Hosts Release CDH Version

dwz-[01-07].nonprod.lan
Supervisord3.0All HostsUnavailableNot applicable
Cloudera Manager Agent6.2.0All Hosts968826.el7Not applicable
Cloudera Manager Management Daemons6.2.0All Hosts968826.el7Not applicable
Flume NG1.9.0+cdh6.2.0All Hosts967373CDH 6
Hadoop3.0.0+cdh6.2.0All Hosts967373CDH 6
HDFS3.0.0+cdh6.2.0All Hosts967373CDH 6
HttpFS3.0.0+cdh6.2.0All Hosts967373CDH 6
hadoop-kms3.0.0+cdh6.2.0All Hosts967373CDH 6
MapReduce 23.0.0+cdh6.2.0All Hosts967373CDH 6
YARN3.0.0+cdh6.2.0All Hosts967373CDH 6
HBase2.1.0+cdh6.2.0All Hosts967373CDH 6
Lily HBase Indexer1.5+cdh6.2.0All Hosts967373CDH 6
Hive2.1.1+cdh6.2.0All Hosts967373CDH 6
HCatalog2.1.1+cdh6.2.0All Hosts967373CDH 6
Hue4.2.0+cdh6.2.0All Hosts967373CDH 6
Impala3.2.0+cdh6.2.0All Hosts967373CDH 6
Java 81.8.0_131All HostsUnavailableNot applicable
Kafka2.1.0+cdh6.2.0All Hosts967373CDH 6
Kite1.0.0+cdh6.2.0All Hosts967373CDH 6
kudu1.9.0+cdh6.2.0All Hosts967373CDH 6
Oozie5.1.0+cdh6.2.0All Hosts967373CDH 6
Parquet1.9.0+cdh6.2.0All Hosts967373CDH 6
Pig0.17.0+cdh6.2.0All Hosts967373CDH 6
sentry2.1.0+cdh6.2.0All Hosts967373CDH 6
Solr7.4.0+cdh6.2.0All Hosts967373CDH 6
spark2.4.0+cdh6.2.0All Hosts967373CDH 6
Sqoop1.4.7+cdh6.2.0All Hosts967373CDH 6
Zookeeper3.4.5+cdh6.2.0All Hosts967373CDH 6

 

Regards,

Sandeep Singh

avatar
Master Guru

@SandeepSingh A quick thing I noticed is that the RHEL7.7 is not supported for the CDH6.2:

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_os_requirements.html#c6...


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar

@GangWar 

Indeed I'm using rhel 7.7.Same behavior is observed on rhel 7.6 as well.

I have performed some tests and everything works smoothly when not enabled SSL on both OS releases (7.6 as well as 7.7).

 

I would guess, setting Enable Kerberos Authentication for HTTP Web-Consoles would kick in only for web-console's like 'Name Node UI etc' and shouldn't have any impact on CM's ability to read service logs from their respective hosts.

 

Regards

Sandeep Singh