HUE 2.3 (CDH 4.3) stopped to show logs for MR jobs and for interqctive pig

[#1]

Hi, I've made a mistake and tried to dump ~100MB to browser window.

I saw a part of output, Then I've realized what I've done and restated hue service

How HUE Stopped to dispaly logs. I see the progress, I see the rolling thing, I don't see any logs on the screen.

HUE service log says:

[13/Aug/2013 12:14:23 +0000] models INFO Retrieving http://prod-node034.lol.ru:50060/tasklog?attemptid=attempt_201307311025_9164_m_000000_0
[13/Aug/2013 12:15:12 +0000] settings INFO Welcome to Hue 2.3.0
[13/Aug/2013 12:15:13 +0000] settings INFO Welcome to Hue 2.3.0
[13/Aug/2013 12:15:41 +0000] middleware INFO Unloading SpnegoMiddleware
[13/Aug/2013 12:15:41 +0000] middleware INFO Unloading HueRemoteUserMiddleware
[13/Aug/2013 12:15:42 +0000] backend INFO Augmenting users with class: <class 'desktop.auth.backend.DefaultUserAugmentor'>
[13/Aug/2013 12:15:42 +0000] access INFO 10.236.34.228 hdfs - "GET / HTTP/1.1"
[13/Aug/2013 12:15:42 +0000] access INFO 10.236.34.228 hdfs - "GET /impala/results/1928/0 HTTP/1.1"
[13/Aug/2013 12:15:42 +0000] thrift_util ERROR Thrift saw exception (this may be expected).
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/share/hue/desktop/core/src/desktop/lib/thrift_util.py", line 338, in wrapper
ret = res(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/share/hue/apps/beeswax/src/beeswax/../../gen-py/TCLIService/TCLIService.py", line 634, in GetLog
return self.recv_GetLog()
File "/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/share/hue/apps/beeswax/src/beeswax/../../gen-py/TCLIService/TCLIService.py", line 650, in recv_GetLog
raise x
TApplicationException: Invalid method name: 'GetLog'
[13/Aug/2013 12:15:42 +0000] thrift_util INFO Thrift saw an application exception: Invalid method name: 'GetLog'
[13/Aug/2013 12:15:43 +0000] access INFO 10.236.34.228 hdfs - "GET /about/ HTTP/1.1"
[13/Aug/2013 12:15:46 +0000] access INFO 10.236.34.228 hdfs - "GET /debug/check_config_ajax HTTP/1.1"
[13/Aug/2013 12:15:55 +0000] access INFO 10.236.34.228 hdfs - "GET /pig/ HTTP/1.1"
[13/Aug/2013 12:15:57 +0000] access INFO 10.236.34.228 hdfs - "GET /debug/check_config_ajax HTTP/1.1"
[13/Aug/2013 12:15:57 +0000] access INFO 10.236.34.228 hdfs - "GET /pig/scripts/ HTTP/1.1"
[13/Aug/2013 12:15:57 +0000] access INFO 10.236.34.228 hdfs - "GET /pig/dashboard/ HTTP/1.1"
[13/Aug/2013 12:16:02 +0000] models INFO Retrieving http://prod-node012.lol.ru:50060/tasklog?attemptid=attempt_201307311025_9188_m_000000_0
[13/Aug/2013 12:16:02 +0000] models INFO Retrieving http://prod-node013.lol.ru:50060/tasklog?attemptid=attempt_201307311025_9115_m_000000_0
[13/Aug/2013 12:16:04 +0000] models WARNING Error parsing task attempt log for attempt_201307311025_9115_m_000000_0 at "http://prod-node013.lol.ru:50060/tasklog?attemptid=attempt_201307311025_9115_m_000000_0". Found 1 (not 3) log sections
[13/Aug/2013 12:16:06 +0000] access INFO 10.236.34.228 hdfs - "GET /pig/dashboard/ HTTP/1.1"

I can easilly access http://prod-node013.lol.ru:50060/tasklog?attemptid=attempt_201307311025_9115_m_000000_0

There is nothing special.

Hoiw can I fix HUE?

[#2]

Time passed and it started to work.
Looks like HUE did clean/rotate something...
I didn't do something special. Magic.

[#3]

The GetLog error is interesting and is likely from the fact that the Beeswax/HS2 contacted by the application wasn't updated to the 4.3 TCLIService spec.

The other error, i.e. the one this report is about, seems to be a failure in parsing the contents somehow. Did this re-occur? Do you use security?

[#4]

1. We are on CDH 4.3 from the first day of official release. There were no problems with logs.

2. No security. We have enough toys to play without security 🙂

3. I think the problem is in some kind of caching on HUE side. I didn't inspect the code, but the problem gone away in few hours without any manual intervention. Looks like persistent cache or stuff like that. Sorry if i'm wrong.

My fisrt action was to restart HUE service. It didn't help. Time passed and the problem disappeared.

[#5]

This remind me of a transient log problem on the hue-user list.

Next time you have something like this:

[13/Aug/2013 12:16:02 +0000] models INFO Retrieving http://prod-node013.lol.ru:50060/tasklog?attemptid=attempt_201307311025_9115_m_000000_0
[13/Aug/2013 12:16:04 +0000] models WARNING Error parsing task attempt log for attempt_201307311025_9115_m_000000_0 at "http://prod-node013.lol.ru:50060/tasklog?attemptid=attempt_201307311025_9115_m_000000_0". Found 1 (not 3) log sections

Could you wget the page and send it that way we can reproduce and fix the problem easily?

[#6 · Accepted solution]

We did have such problems before. Our guys were debugging pig script which output is html/json. It did break HUE. Restart helped immediately.

We've already reported it in HUE group. And I did provide wget as I remember.

The new problem (described here) is someting new. The output are plain old pig tuples with numbers.

Next time I'll provide wget. Sorry, I forgot to do it this time.

I suggest to close this topic because I can't provide data to reproduce issue.

[#7]

I remember this, but the wgeted files was parsed proprely for me (2.3).

In the Pig case, maybe the Pig output broke the parsing but yes, getting the output would be great in order to reproduce and fix it.