Reply
New Contributor
Posts: 8
Registered: ‎05-03-2018

Authentication token expired at impala coordinators after Kudu restart

[ Edited ]

Hi,

 

The issue (see log below) causes to be unable to launch impala query on Kudu tables through coordinator (but launch through executors work fine).

Appeared after Kudu (masters and tservers) was restarted.

CDH 5.14.2.

LDAP authentication is enabled for impala coordinator nodes only.

 

Let me know if any additional information is needed.

Any ideas how to avoid the issue?

 

Message in impalad log at impala coordinator daemon:

Tuple(id=0 size=49 slots=[Slot(id=0 type=STRING col_path=[] offset=0 null=(offset=48 mask=1) slot_idx=0 field_idx=-1), Slot(id=1 type=STRING col_path=[] offset=16 null=(offset=48 mask=2) slot_idx=1 field_idx=-1), Slot(id=2 type=STRING col_path=[] offset=32 null=(offset=48 mask=4) slot_idx=2 field_idx=-1)] tuple_path=[])
I0503 09:29:01.486366 41053 coordinator.cc:370] started execution on 1 backends for query 964f6d68c729a5cd:2ad2ba7200000000
I0503 09:29:01.486568 42404 query-state.cc:377] Executing instance. instance_id=964f6d68c729a5cd:2ad2ba7200000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=2
I0503 09:29:01.487128 42403 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=964f6d68c729a5cd:2ad2ba7200000000 refcnt=3
I0503 09:29:01.487524 41053 impala-hs2-server.cc:492] ExecuteStatement(): return_val=TExecuteStatementResp {
01: status (struct) = TStatus {
01: statusCode (i32) = 0,
},
02: operationHandle (struct) = TOperationHandle {
01: operationId (struct) = THandleIdentifier {
01: guid (string) = "<some value>",
02: secret (string) = "<some value>",
},
02: operationType (i32) = 0,
03: hasResultSet (bool) = false,
},
}
I0503 09:29:01.487515 42406 coordinator.cc:789] Coordinator waiting for backends to finish, 1 remaining
I0503 09:29:01.491019 42404 client-internal.cc:281] Determining the new leader Master and retrying...
I0503 09:29:01.507668 42404 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to <Kudu_leader_IP>:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired
I0503 09:29:01.528237 42404 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to <Kudu_leader_IP>:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired
...

 

 

 

Cloudera Employee
Posts: 10
Registered: ‎02-22-2017

Re: Authentication token expired at impala coordinators after Kudu restart

Hi Andreyeff,

 

'FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired' error indicates this is authn token expiration issue. However, in CDH5.14.2, Kudu client should be able to automatically re-acquires authn token when needed.  Do you know which Kudu client version you are using (CDH5.14.2)?  When you launch the impala query, does the coordinator have primary credentials (i.e. Kerberos)?

 

Best,

Hao

New Contributor
Posts: 8
Registered: ‎05-03-2018

Re: Authentication token expired at impala coordinators after Kudu restart

[ Edited ]

Answering your questions:

1) kudu -version
kudu 1.6.0-cdh5.14.2

However, not sure if Kudu client is used in communication between Impala and Kudu.

2) Impala coordinators use LDAP authentication, impala executors - don't.

Kerberos is not used.

Moreover, the new session is initiated and the same issue appears (this happens till Impala is restarted).

 

The sequence was:

Normally the cluster works, Impala-users connect and execute queries through coordinator.

There was Kudu leader change from one master to another (possibly also masterA->masterC->masterA).

The new Impala-user runs a query and receives this issue.

Impala restart has removed the effect and cluster returned to usual operating mode.

Cloudera Employee
Posts: 36
Registered: ‎04-08-2014

Re: Authentication token expired at impala coordinators after Kudu restart

Is it possible that Impala was not restarted after a previous upgrade?

 

Please report the issue if this happens again on CDH 5.14+

 

Thanks,

Mike

New Contributor
Posts: 8
Registered: ‎05-03-2018

Re: Authentication token expired at impala coordinators after Kudu restart

The cluster was stopped during upgrade (from 5.13.0), so impala (as well as other services) is expected to be started with new configs since then.

 

From cluster inspect the only thing remained running from 5.13 is supervisord process.

 

I'll share more details if it appears again.

Cloudera Employee
Posts: 36
Registered: ‎04-08-2014

Re: Authentication token expired at impala coordinators after Kudu restart

Please do. Thanks in advance for that.

Announcements