Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Not able to view yarn and JH logs in Knox

I am using knox 0.12.0 and configured topology for YARNUI to be displayed by knox URL. The initial webpage loads and yarn applications are visible, but when I click on the logs link there is an error like "Problem accessing /node/containerlogs/container_e17_1495012138865_0004_01_000001/hive/stderr. Reason:"Not Found. how to display the yarn logs on the browser using knox. Similar is the case with jobhistory server, error is "Cannot get container logs. Invalid nodeId: datahost.ec2.internal%3A45454".

7 REPLIES 7

Contributor

Hello @Sanjeev Rao, Can you paste relevant Knox log snippet, that would make it a bit more clear.

Knox Logs looks like this:

For Jobhistory:

17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI||||access|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|unavailable|Request method: GET 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authentication|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success| 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authentication|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success|Groups: [] 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authorization|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success| 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||dispatch|uri|http://ip-10-228-3-43.ec2.internal:19888/jobhistory/logs/ip-10-228-1-83.ec2.internal%3A45454/contain... method: GET 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||dispatch|uri|http://ip-10-228-3-43.ec2.internal:19888/jobhistory/logs/ip-10-228-1-83.ec2.internal%3A45454/contain... status: 200 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||access|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success|Response status: 200

Yarn:

17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI||||access|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|unavailable|Request method: GET 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authentication|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success| 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authentication|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success|Groups: [] 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authorization|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success| 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||dispatch|uri|http://10.228.2.223:8088/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr?u... method: GET 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||dispatch|uri|http://10.228.2.223:8088/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr?u... status: 404 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||access|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success|Response status: 404

yarn browser image as below:

15546-capture.png

Jobhistory UI output as below:

15547-capture1.png

@Sandeep More please let me know if you need further information.

Contributor

You are getting a 404, so there could two possibilities a) the service might be down or node not accessible b) Knox might not be rewriting the URL correctly.

Do you have the output for gateway.log ? this looks like gateway-audit.log output.

The gateway.log looks like this:

2017-05-18 09:27:40,061 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,065 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,088 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,096 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,100 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,102 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true

New Contributor

For the first case, if it is a running job, the error is likely because the "host=<node-manager>" query parameter is missing from the URL. The rewrite rule that gets applied is this:

2017-06-27 05:10:28,138 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: /node/containerlogs/container_e01_1498485266952_0006_01_000001/hiyer qa3july/stderr/?start=-4096, direction: OUT via implicit rule: YARNUI/yarn/outbound/node to URL: https://host:8443/gateway/sandbox/yarn/node/containerlogs/container_e01_1498485266952_0006_01_000001...

The correct URL should have a "?host=<nodemanager>" at the end.

Unfortunately I've not figured out a way to work around this yet 😞

Expert Contributor

@Sanjeev Rao check this - https://issues.apache.org/jira/browse/KNOX-728. This could be related.