question Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster in Support Questions

"No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

SandeepSingh — Tue, 15 Oct 2019 14:37:58 GMT

Hi All,

When Kerberos and TLS is enabled on the CDH cluster, role logs are not reported in Cloudera Manager under the respective services. CM reports 'No log messages at the specified URL' for each of the following:

Role Log File
Stdout
Stderr

Interestingly, the log files mentioned on the log pages (on CM) exist on the underlined host and contain all & correct log entries; They just don't get reported at CM service log pages. When tried downloading full log files, it throws error:

HTTP ERROR 403

Problem accessing /cmf/process/335/logs. Reason:

    Unexpected end of file from server
The server declined access to the page or resource.

Powered by Jetty:// 9.4.14.v20181114

Same error is observed when attempted fetching logs using CM API.

Environment:

CDH	6.2
OS	Redhat 7.7

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

Bhuv — Wed, 16 Oct 2019 10:49:10 GMT

Hi Sandeep,

Can you please check the owner and permissions of "/var/run/cloudera-scm-server" properties whether it is root or cloudera-scm ??

Thanks and Regards,
Bhuvan.

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

SandeepSingh — Wed, 16 Oct 2019 11:09:13 GMT

Hi @Bhuv

Here are the owner/permission details:

drwxr-xr-x. 2 cloudera-scm cloudera-scm 40 Sep 24 12:05 cloudera-scm-server

Regards,

Sandeep

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

GangWar — Thu, 17 Oct 2019 18:16:22 GMT

@SandeepSingh If the same error is also can be observed from the API then this is most probably the issue with TLS. However could you please try to narrow down the issue by following methods:

Run the following command on every Agent host with supervisorctl:

/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

If Enable Kerberos Authentication for HTTP Web-Consoles is checked in respective service then please configure SPNEGO for browsers: https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_browser_access_kerberos_protected_url.html

Please let us know how this goes. If the issue remains then please run the Host Inspector and share the result.

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

lwang — Thu, 17 Oct 2019 21:09:30 GMT

Hi @SandeepSingh ,

Could you please share your Cloudera Manager version? I see you mentioned CDH version is 6.2. What about CM version? There was a bug in CM 6.1 where under certain condition the CM Server fails to contact agent when TLS is on. I am wondering if you are hitting that bug.

Thanks,

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

SandeepSingh — Fri, 18 Oct 2019 16:00:00 GMT

Hi @lwang

CM version in this case is 6.2, so I suspect if I am hitting the bug you've mentioned; Unless this wasn't fixed in 6.2 too. Do you have an info?

Regards
Sandeep

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

GangWar — Fri, 18 Oct 2019 16:14:39 GMT

@SandeepSingh Have you tried to run below command in problematic host as stated earlier?

/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

This bug has been fixed in the 6.2. I may be possible the status server did not being restarted after making the changes. Please run the command and also follow the other steps stated in previous post and let us know how this goes.

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

SandeepSingh — Wed, 23 Oct 2019 12:49:24 GMT

Hi @GangWar

I tried the both of the following command; however result was exactly the same.

/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

I have indeed Enable Kerberos Authentication for HTTP Web-Consoles option checked-in and I already tried configuring SPNEGO in the browsers (Mozilla Firefox as well Chrome) as stated in Cloudera documentation; no luck.

Host Inspector also failed to complete on all hosts of the cluster. Error Details:

IOException thrown while collecting data from host: Read timed out

stdout
 "etcKrbConfMessages" : [ ],
  "extantInitdErrors" : [ ],
  "groupData" : "hdfs:x:11004:hive,impala\nmapred:x:11014:\nzookeeper:x:11022:\noozie:x:11015:\nhbase:x:11003:\nhue:x:11007:\ncloudera-scm:x:991:\nhadoop:x:11002:hdfs,hive,impala,mapred,yarn\nhive:x:11005:impala\nsqoop:x:11019:\nimpala:x:11008:\nyarn:x:11021:\nhttpfs:x:11006:\nsentry:x:11018:\nsolr:x:11016:\nspark:x:11017:\nkms:x:11010:\n",
  "hostDnsErrors" : [ ],
  "hostname" : "dwz-06.nonprod.lan",
  "jceStrength" : 0,
  "kdcConnectionMessages" : [ ],
  "kernelVersion" : "3.10.0-1062.1.2.el7.x86_64",
  "kernelVersionException" : null,
  "localHostIpError" : null,
  "localhostIp" : "127.0.0.1",
  "nowMillis" : 1571833401403,
  "psycopg2Version" : "2.7.5",
  "psycopg2VersionException" : null,
  "psycopg2VersionOk" : true,
  "pythonVersionException" : null,
  "pythonVersionOk" : true,
  "pythonVersionString" : "2.7",
  "rhelRelease" : "Red Hat Enterprise Linux Server release 7.7 (Maipo)",
  "runExceptions" : [ ],
  "swappiness" : "60",
  "swappinessException" : null,
  "timeZone" : "UTC+01:00",
  "transparentHugePagesDefrag" : "[always] madvise never",
  "transparentHugePagesEnabled" : "always madvise [never]",
  "transparentHugePagesException" : null,
  "transparentHugePagesPath" : "/sys/kernel/mm/transparent_hugepage",
  "userData" : "hdfs:x:980:11004::/home/hdfs:/bin/bash\nmapred:x:977:11014::/home/mapred:/bin/bash\nzookeeper:x:981:11022::/home/zookeeper:/bin/bash\noozie:x:986:11015::/home/oozie:/bin/bash\nhbase:x:994:11003::/home/hbase:/bin/bash\nhue:x:992:11007::/home/hue:/bin/bash\ncloudera-scm:x:974:991:Cloudera Manager:/var/lib/cloudera-scm-server:/sbin/nologin\nsqoop:x:982:11019::/home/sqoop:/bin/bash\nsqoop2:x:976:11019::/home/sqoop2:/bin/bash\nimpala:x:978:11008::/home/impala:/bin/bash\nyarn:x:975:11021::/home/yarn:/bin/bash\nhttpfs:x:993:11006::/home/httpfs:/bin/bash\nsentry:x:983:11018::/home/sentry:/bin/bash\nsolr:x:985:11016::/home/solr:/bin/bash\nspark:x:984:11017::/home/spark:/bin/bash\nkms:x:990:11010::/home/kms:/bin/bash\n"
}

stderr
+ [[ inspector == \f\i\r\e\h\o\s\e ]]
+ [[ inspector == \e\v\e\n\t\s\e\r\v\e\r ]]
+ [[ inspector == \a\l\e\r\t\p\u\b\l\i\s\h\e\r ]]
+ [[ inspector == \h\e\a\d\l\a\m\p ]]
+ [[ inspector == \t\e\l\e\m\e\t\r\y\p\u\b\l\i\s\h\e\r ]]
+ [[ inspector == \t\e\s\t\-\d\b\u\s\-\c\o\n\n\e\c\t\i\o\n ]]
+ [[ inspector == \i\n\s\p\e\c\t\o\r ]]
+ shift
++ pwd
+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ echo_and_exec /usr/java/jdk1.8.0_131/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ echo 'Executing: /usr/java/jdk1.8.0_131/bin/java' -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ exec /usr/java/jdk1.8.0_131/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/86-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT

Here is what CM shows on Host Inspection result page:

Validations

	Inspector failed on the following hosts... View Details dwz-[01-07].nonprod.lan: IOException thrown while collecting data from host: Read timed out Inspector ran on 0 hosts.
	The inspector failed to run on all hosts.
	0 hosts are running CDH 5 and 7 hosts are running CDH 6.
	All checked hosts in each cluster are running the same version of components.
	All managed hosts have consistent versions of Java.
	All checked Cloudera Management Daemons versions are consistent with the server.
	All checked Cloudera Management Agents versions are consistent with the server.

Version Summary

AfisPOCDataLake

All HostsComponent Version Hosts Release CDH Version

dwz-[01-07].nonprod.lan
Supervisord	3.0	All Hosts	Unavailable	Not applicable
Cloudera Manager Agent	6.2.0	All Hosts	968826.el7	Not applicable
Cloudera Manager Management Daemons	6.2.0	All Hosts	968826.el7	Not applicable
Flume NG	1.9.0+cdh6.2.0	All Hosts	967373	CDH 6
Hadoop	3.0.0+cdh6.2.0	All Hosts	967373	CDH 6
HDFS	3.0.0+cdh6.2.0	All Hosts	967373	CDH 6
HttpFS	3.0.0+cdh6.2.0	All Hosts	967373	CDH 6
hadoop-kms	3.0.0+cdh6.2.0	All Hosts	967373	CDH 6
MapReduce 2	3.0.0+cdh6.2.0	All Hosts	967373	CDH 6
YARN	3.0.0+cdh6.2.0	All Hosts	967373	CDH 6
HBase	2.1.0+cdh6.2.0	All Hosts	967373	CDH 6
Lily HBase Indexer	1.5+cdh6.2.0	All Hosts	967373	CDH 6
Hive	2.1.1+cdh6.2.0	All Hosts	967373	CDH 6
HCatalog	2.1.1+cdh6.2.0	All Hosts	967373	CDH 6
Hue	4.2.0+cdh6.2.0	All Hosts	967373	CDH 6
Impala	3.2.0+cdh6.2.0	All Hosts	967373	CDH 6
Java 8	1.8.0_131	All Hosts	Unavailable	Not applicable
Kafka	2.1.0+cdh6.2.0	All Hosts	967373	CDH 6
Kite	1.0.0+cdh6.2.0	All Hosts	967373	CDH 6
kudu	1.9.0+cdh6.2.0	All Hosts	967373	CDH 6
Oozie	5.1.0+cdh6.2.0	All Hosts	967373	CDH 6
Parquet	1.9.0+cdh6.2.0	All Hosts	967373	CDH 6
Pig	0.17.0+cdh6.2.0	All Hosts	967373	CDH 6
sentry	2.1.0+cdh6.2.0	All Hosts	967373	CDH 6
Solr	7.4.0+cdh6.2.0	All Hosts	967373	CDH 6
spark	2.4.0+cdh6.2.0	All Hosts	967373	CDH 6
Sqoop	1.4.7+cdh6.2.0	All Hosts	967373	CDH 6
Zookeeper	3.4.5+cdh6.2.0	All Hosts	967373	CDH 6

Regards,

Sandeep Singh

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

GangWar — Thu, 24 Oct 2019 18:32:52 GMT

@SandeepSingh A quick thing I noticed is that the RHEL7.7 is not supported for the CDH6.2:

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_os_requirements.html#c62_supported_os

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

SandeepSingh — Tue, 29 Oct 2019 16:28:50 GMT

@GangWar

Indeed I'm using rhel 7.7.Same behavior is observed on rhel 7.6 as well.

I have performed some tests and everything works smoothly when not enabled SSL on both OS releases (7.6 as well as 7.7).

I would guess, setting Enable Kerberos Authentication for HTTP Web-Consoles would kick in only for web-console's like 'Name Node UI etc' and shouldn't have any impact on CM's ability to read service logs from their respective hosts.

Regards

Sandeep Singh

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

GangWar — Thu, 28 Nov 2019 17:28:36 GMT

@SandeepSingh This looks like the issue with TLS.

Eventhough the flag 'Use TLS Authentication of Agents to Server' in CM WebUI is not set, the following flag must be set for status_server to use TLS protocol using port 9000. Go to the /opt/cloudera/security/x509/ directory and use 'pem' and 'key' file under that directory. You may also have to use the password file for the private key if there is one.

Then edit the /etc/cloudera-scm-agent/config.ini file with below parameters.

# PEM file containing client private key.
client_key_file=

# If client_keypw_cmd isn't specified, instead a text file containing the client private key password can be used.
client_keypw_file=

# PEM file containing client certificate.
client_cert_file=/etc/cdep-ssl-conf/CA_STANDARD/cm_server-cert.pem

verify_cert_file=

Restart of the status_server is required

cd /var/run/cloudera-scm-agent/supervisord
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

In addition, restart of the cloudera-scm-agent is also needed
service cloudera-scm-agent restart

Re: "No log messages at the specified URL" for Role Logs on Kerberized and TLS enabled cluster

SandeepSingh — Wed, 11 Mar 2020 16:26:21 GMT

@GangWar Thanks for your suggestion.

All the parameters except the following one were already set in /etc/cloudera-scm-agent/config.ini

verify_cert_file

Apparently, the only reason why agent wasn't serving requests for logs was because the above flat wasn't set.

The moment we configured the flag verify_cert_file and restarted agent, it started serving logs correctly.