Support Questions
Find answers, ask questions, and share your expertise

Resourcemanager spams krb5kdc.log

Highlighted

Resourcemanager spams krb5kdc.log

Contributor

Hello,

I'm wondering if anyone has any advice regarding this problem:

Our krb5kdc.log is filling up with lines like these:

Jan 01 03:18:02 censored-hostname2 krb5kdc[66490](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 10.204.167.8: ISSUE: authtime 1451550638, etypes {rep=18 tkt=18 ses=18}, rm/censored-hostname@OUR.REALM for HTTP/censored-hostname@OUR.REALM

The current file contains log messages from 4.5 days and I have about 4 million of these lines in that file. These are filling up the /var partition for us.

I can't find anything in the resourcemanager logs that indicates that it would spam the kdc with ticket requests.

We are running HDP-2.2.6.0 and MIT Kerberos. The cluster is used in production and is at times heavily utilized.

Grateful for any input.

/Thomas

8 REPLIES 8
Highlighted

Re: Resourcemanager spams krb5kdc.log

@Thomas Larsson What HDP version? and Re: RM , Is this happening in idle cluster? MIT KDC?

Highlighted

Re: Resourcemanager spams krb5kdc.log

Contributor

@Neeraj Sabharwal

I've updated the description with the additional info.

Highlighted

Re: Resourcemanager spams krb5kdc.log

@Thomas Larsson

MIT logging level....You may want to check that and also, place log rotation in place after backing up existing log

http://web.mit.edu/kerberos/krb5-1.4/krb5-1.4.1/doc/krb5-admin/logging.html

SYSLOG[:<severity>[:<facility>]]
This causes the entity's logging messages to go to the system log.

The severity argument specifies the default severity of system log messages. This may be any of the following severities supported by the syslog(3) call, minus the LOG_ prefix: LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR, LOG_WARNING, LOG_NOTICE, LOG_INFO, and LOG_DEBUG. For example, a value of CRIT would specify LOG_CRIT severity.

The facility argument specifies the facility under which the messages are logged. This may be any of the following facilities supported by the syslog(3) call minus the LOG_ prefix: LOG_KERN, LOG_USER, LOG_MAIL, LOG_DAEMON, LOG_AUTH, LOG_LPR, LOG_NEWS, LOG_UUCP, LOG_CRON, and LOG_LOCAL0 through LOG_LOCAL7.

Re: Resourcemanager spams krb5kdc.log

Contributor

@Neeraj Sabharwal

Thanks Neeraj. We have implemented stricter log retention but I feel that is just sweeping the issue under the carpet. I still think it's weird that the resourcemanager of our small cluster (I forgot to mention, its about 20 large datanodes) should produce more than 10 ticket requests per second like this. Do you agree or is just as expected?

Highlighted

Re: Resourcemanager spams krb5kdc.log

Highlighted

Re: Resourcemanager spams krb5kdc.log

Contributor

Ok. Thanks @Neeraj Sabharwal. The error messages in that link "Server not found in database" is not something I see in our logs. For now, we will stick with just the stricter log retention unless we see worse problems. I'll report back if we find out the reason. Thanks again.

Highlighted

Re: Resourcemanager spams krb5kdc.log

@Thomas Larsson I suggest to open a support ticket for further analysis.

Highlighted

Re: Resourcemanager spams krb5kdc.log

Mentor

@Thomas Larsson has this been resolved? Please accept best answer or provide your own solution.