Cannot read log files on applications started by LDAP users in YARN

BrightLiao — Tue, 19 Jan 2021 09:34:32 GMT

When I try to look at logs from command `yarn logs -applicationId xxx`, some errors raised:

Response from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:
Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : syslog
Can not get any logs for the log file: stderr
Response from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:
Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : stderr
Can not get any logs for the log file: prelaunch.out
Response from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:
Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : prelaunch.out
Can not get any logs for the log file: launch_container.sh

The question is similar to https://community.cloudera.com/t5/Support-Questions/Cannot-read-log-files-on-applications-started-by-Active/td-p/214539.

But the exception is another one. From the node manager log, the error is something like:

java.io.IOException: Owner 'infra-solr' for path /data02/hadoop/yarn/log/application_1610695610027_0516/container_e
11_1610695610027_0516_01_000002/prelaunch.out did not match expected owner 'xxx'
at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:281)
at org.apache.hadoop.io.SecureIOUtils.forceSecureOpenForRead(SecureIOUtils.java:215)
at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:200)
at org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerLogsUtils.openLogFileForRead(ContainerLogsUtil
s.java:169)
at org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices.getLogs(NMWebServices.java:467)
at org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices.getContainerLogFile(NMWebServices.java:38
0)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$Response

My question is that why the log file owner is 'infra-solr', is it copied by ‘infra-solr’ from somewhere else?

Hope someone could help, Thanks!

Re: Cannot read log files on applications started by LDAP users in YARN

BrightLiao — Wed, 20 Jan 2021 03:10:33 GMT

It's the same LDAP uid conflicts issue with this one: https://community.cloudera.com/t5/Support-Questions/Hive-tez-vertex-failed-during-map-phase-HDP-3-1-4/td-p/309973

question Cannot read log files on applications started by LDAP users in YARN in Support Questions

Cannot read log files on applications started by LDAP users in YARN

Re: Cannot read log files on applications started by LDAP users in YARN