Reply
Explorer
Posts: 7
Registered: ‎08-12-2016

Yarn Resource Manager warnings in logs regarding Authentication token

[ Edited ]

Hi, 

 

Does any one know why i am seeing these warnings in my Resource manager logs ? 

 

These are coming every minute and thats why they are concerning, there were no changes made lately to the cluster. Could somoene throw some light on these errors and point me to the right direction for further investigation. 

 

IMPORTANT  AuthenticationToken ignored: Invalid AuthenticationToken type

Service: YARN (MR2 Included)  Role: ResourceManager (xxxx)  Hosts: xxx.com  Category: LOG_MESSAGE   Collapse

CATEGORY

LOG_MESSAGE

CLUSTER

cluster

CLUSTER_DISPLAY_NAME

PROD

CLUSTER_ID

6

EVENTCODE

EV_LOG_EVENT

HOSTS

xxx.com

HOST_IDS

4b944d35-8be8-4b58-8ccd-xxxxxxxxx

LOG_LEVEL

WARN

ROLE

ResourceManager (xxxxx)

ROLE_TYPE

RESOURCEMANAGER

SERVICE

YARN (MR2 Included)

SERVICE_TYPE

YARN

SEVERITY

IMPORTANT

URL

http://xxxxxx:7180/cmf/eventRedirect/6ef48809-2126-42b6-a5e5-1d83d7efdff2

__persist_timestamp

1491931149439

__uuid

6ef48809-2126-42b6-a5e5-1xxxxxxxx

Posts: 332
Topics: 0
Kudos: 34
Solutions: 26
Registered: ‎08-16-2016

Re: Yarn Resource Manager warnings in logs regarding Authentication token

I would drilldown into the RM logs themselves. this is just an even message. The logs should have more info and maybe an exception.

Is Kerberos enabled on the cluster?
Explorer
Posts: 7
Registered: ‎08-12-2016

Re: Yarn Resource Manager warnings in logs regarding Authentication token

ok. will look further into the RM logs. 

 

Yes Kerberos is enabled on the cluster. 

Explorer
Posts: 7
Registered: ‎08-12-2016

Re: Yarn Resource Manager warnings in logs regarding Authentication token

I also see these warnings as well.

“PriviledgedActionException as:mapred/xxxxx@xxxx.COM (auth:KERBEROS) cause:org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1489952554624_125489' doesn't exist in RM.”

Not sure if they are related.
Posts: 332
Topics: 0
Kudos: 34
Solutions: 26
Registered: ‎08-16-2016

Re: Yarn Resource Manager warnings in logs regarding Authentication token

What service, roles, or host is the last message coming from? The only service that I know of that runs under the mapred account is the JobHistory server. This could mean there is an issue with jobs transitioning between the RM and JH when they are aged out.

The first on is not very helpful, but it could indicate that some user/service is trying to interact with the RM but has a bad token so it fails.

Announcements