Not able to view yarn and JH logs in Knox

[#1]

I am using knox 0.12.0 and configured topology for YARNUI to be displayed by knox URL. The initial webpage loads and yarn applications are visible, but when I click on the logs link there is an error like "Problem accessing /node/containerlogs/container_e17_1495012138865_0004_01_000001/hive/stderr. Reason:"Not Found. how to display the yarn logs on the browser using knox. Similar is the case with jobhistory server, error is "Cannot get container logs. Invalid nodeId: datahost.ec2.internal%3A45454".

[#2]

Hello @Sanjeev Rao, Can you paste relevant Knox log snippet, that would make it a bit more clear.

[#3]

Knox Logs looks like this:

For Jobhistory:

17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI||||access|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|unavailable|Request method: GET 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authentication|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success| 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authentication|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success|Groups: [] 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||authorization|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success| 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||dispatch|uri|http://ip-10-228-3-43.ec2.internal:19888/jobhistory/logs/ip-10-228-1-83.ec2.internal%3A45454/contain... method: GET 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||dispatch|uri|http://ip-10-228-3-43.ec2.internal:19888/jobhistory/logs/ip-10-228-1-83.ec2.internal%3A45454/contain... status: 200 17/05/18 04:27:56 ||84a60c35-a082-4c94-82df-12e26cdea8bc|audit|JOBHISTORYUI|guest|||access|uri|/gateway/default/jobhistory/joblogs/ip-10-228-1-83.ec2.internal:45454/container_e19_1495094385014_0002_01_000002/attempt_1495094385014_0002_m_000000_0/ambari-qa|success|Response status: 200

Yarn:

17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI||||access|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|unavailable|Request method: GET 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authentication|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success| 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authentication|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success|Groups: [] 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||authorization|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success| 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||dispatch|uri|http://10.228.2.223:8088/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr?u... method: GET 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||dispatch|uri|http://10.228.2.223:8088/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr?u... status: 404 17/05/18 06:05:53 ||488b9b3c-885d-4b79-a035-385ccded234d|audit|YARNUI|guest|||access|uri|/gateway/default/yarn/node/containerlogs/container_e19_1495094385014_0004_01_000690/hive/stderr|success|Response status: 404

yarn browser image as below:

Jobhistory UI output as below:

[#4]

@Sandeep More please let me know if you need further information.

[#5]

You are getting a 404, so there could two possibilities a) the service might be down or node not accessible b) Knox might not be rewriting the URL correctly.

Do you have the output for gateway.log ? this looks like gateway-audit.log output.

[#6]

The gateway.log looks like this:

2017-05-18 09:27:40,061 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,065 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,088 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,096 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,100 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-05-18 09:27:40,102 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true

[#7]

For the first case, if it is a running job, the error is likely because the "host=<node-manager>" query parameter is missing from the URL. The rewrite rule that gets applied is this:

2017-06-27 05:10:28,138 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: /node/containerlogs/container_e01_1498485266952_0006_01_000001/hiyer qa3july/stderr/?start=-4096, direction: OUT via implicit rule: YARNUI/yarn/outbound/node to URL: https://host:8443/gateway/sandbox/yarn/node/containerlogs/container_e01_1498485266952_0006_01_000001...

The correct URL should have a "?host=<nodemanager>" at the end.

Unfortunately I've not figured out a way to work around this yet 😞

[#8]

@Sanjeev Rao check this - https://issues.apache.org/jira/browse/KNOX-728. This could be related.