Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Not able to view yarn and JH logs in Knox

Highlighted

Not able to view yarn and JH logs in Knox

New Contributor

I am using knox 0.12.0 and configured topology for YARNUI to be displayed by knox URL. The initial webpage loads and yarn applications are visible, but when I click on the logs link there is an error like "Problem accessing /node/containerlogs/container_e17_1495012138865_0004_01_000001/hive/stderr. Reason:"Not Found. how to display the yarn logs on the browser using knox. Similar is the case with jobhistory server, error is "Cannot get container logs. Invalid nodeId: datahost.ec2.internal%3A45454".

7 REPLIES 7

Re: Not able to view yarn and JH logs in Knox

Contributor

Hello @Sanjeev Rao, Can you paste relevant Knox log snippet, that would make it a bit more clear.

Re: Not able to view yarn and JH logs in Knox

New Contributor

Knox Logs looks like this:

For Jobhistory:

17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI||||access|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|unavailable|Request method: GET 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authentication|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success| 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authentication|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success|Groups: [] 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authorization|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success| 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||dispatch|uri|http://ip-10-228-3-43.ec2.internal:19888/jobhistory/logs/ip-10-228-1-83.ec2.internal%3A45454/contain... method: GET 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||dispatch|uri|http://ip-10-228-3-43.ec2.internal:19888/jobhistory/logs/ip-10-228-1-83.ec2.internal%3A45454/contain... status: 200 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||access|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success|Response status: 200

Yarn:

17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI||||access|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|unavailable|Request method: GET 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authentication|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success| 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authentication|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success|Groups: [] 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authorization|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success| 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||dispatch|uri|http://10.228.2.223:8088/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr?u... method: GET 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||dispatch|uri|http://10.228.2.223:8088/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr?u... status: 404 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||access|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success|Response status: 404

yarn browser image as below:

15546-capture.png

Jobhistory UI output as below:

15547-capture1.png

Re: Not able to view yarn and JH logs in Knox

New Contributor

@Sandeep More please let me know if you need further information.

Re: Not able to view yarn and JH logs in Knox

Contributor

You are getting a 404, so there could two possibilities a) the service might be down or node not accessible b) Knox might not be rewriting the URL correctly.

Do you have the output for gateway.log ? this looks like gateway-audit.log output.

Re: Not able to view yarn and JH logs in Knox

New Contributor

The gateway.log looks like this:

2017-05-18 09:27:40,061 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,065 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,088 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,096 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,100 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,102 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true

Re: Not able to view yarn and JH logs in Knox

New Contributor

For the first case, if it is a running job, the error is likely because the "host=<node-manager>" query parameter is missing from the URL. The rewrite rule that gets applied is this:

2017-06-27 05:10:28,138 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: /node/containerlogs/container_e01_1498485266952_0006_01_000001/hiyer qa3july/stderr/?start=-4096, direction: OUT via implicit rule: YARNUI/yarn/outbound/node to URL: https://host:8443/gateway/sandbox/yarn/node/containerlogs/container_e01_1498485266952_0006_01_000001...

The correct URL should have a "?host=<nodemanager>" at the end.

Unfortunately I've not figured out a way to work around this yet :-(

Re: Not able to view yarn and JH logs in Knox

Expert Contributor

@Sanjeev Rao check this - https://issues.apache.org/jira/browse/KNOX-728. This could be related.

Don't have an account?
Coming from Hortonworks? Activate your account here