Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Resource Manager warnings in logs regarding Authentication token

Yarn Resource Manager warnings in logs regarding Authentication token

Explorer

Hi, 

 

Does any one know why i am seeing these warnings in my Resource manager logs ? 

 

These are coming every minute and thats why they are concerning, there were no changes made lately to the cluster. Could somoene throw some light on these errors and point me to the right direction for further investigation. 

 

IMPORTANT  AuthenticationToken ignored: Invalid AuthenticationToken type

Service: YARN (MR2 Included)  Role: ResourceManager (xxxx)  Hosts: xxx.com  Category: LOG_MESSAGE   Collapse

CATEGORY

LOG_MESSAGE

CLUSTER

cluster

CLUSTER_DISPLAY_NAME

PROD

CLUSTER_ID

6

EVENTCODE

EV_LOG_EVENT

HOSTS

xxx.com

HOST_IDS

4b944d35-8be8-4b58-8ccd-xxxxxxxxx

LOG_LEVEL

WARN

ROLE

ResourceManager (xxxxx)

ROLE_TYPE

RESOURCEMANAGER

SERVICE

YARN (MR2 Included)

SERVICE_TYPE

YARN

SEVERITY

IMPORTANT

URL

http://xxxxxx:7180/cmf/eventRedirect/6ef48809-2126-42b6-a5e5-1d83d7efdff2

__persist_timestamp

1491931149439

__uuid

6ef48809-2126-42b6-a5e5-1xxxxxxxx

4 REPLIES 4

Re: Yarn Resource Manager warnings in logs regarding Authentication token

Champion
I would drilldown into the RM logs themselves. this is just an even message. The logs should have more info and maybe an exception.

Is Kerberos enabled on the cluster?

Re: Yarn Resource Manager warnings in logs regarding Authentication token

Explorer

ok. will look further into the RM logs. 

 

Yes Kerberos is enabled on the cluster. 

Re: Yarn Resource Manager warnings in logs regarding Authentication token

Explorer
I also see these warnings as well.

“PriviledgedActionException as:mapred/xxxxx@xxxx.COM (auth:KERBEROS) cause:org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1489952554624_125489' doesn't exist in RM.”

Not sure if they are related.

Re: Yarn Resource Manager warnings in logs regarding Authentication token

Champion

What service, roles, or host is the last message coming from? The only service that I know of that runs under the mapred account is the JobHistory server. This could mean there is an issue with jobs transitioning between the RM and JH when they are aged out.

The first on is not very helpful, but it could indicate that some user/service is trying to interact with the RM but has a bad token so it fails.

Don't have an account?
Coming from Hortonworks? Activate your account here