Kudu and Kerberos

[#1]

Hi,

I’m on CDH 5.13.0 with kudu 1.5.0. I have a problem when i enable kerberos authentication on kudu. Kerberos authentication work fine on other components (hbase, hdfs, impala, etc…).

When I try to create a table on kudu storage with hue or impala-shell, I have an error 

Query :

create table kudu_db.test3 (

row_id string,

test string,

primary key (row_id)

)

partition by hash (row_id) partitions 8

stored as kudu

Error :

ImpalaRuntimeException: Error creating Kudu table 'impala::s4do05k0_p04.test3' CAUSED BY: NonRecoverableException: Not enough live tablet servers to create a table with the requested replication factor 3. 0 tablet servers are alive.

In cloudera manager/Kudu Master Web UI/ « Tablet Servers » tab, i have this :

If I disable kerberos, I have this :

 

Configuration in Cloudera Manager

Create table doesn’t work but I can select on existing table...

Anyone can help me please ?

Best regards

[#2]

Hi,

Can you check what keytab your tablet servers are running with?

You can do that by logging in to one of the tablet server machines and checking the command line that kudu-tserver process is running with.   Then check what's inside that keytab.

It's something like

[root@anonymous ~]# ps axw | grep kudu-tserver

  548 pts/0    S+     0:00 grep --color=auto kudu-tserver

32747 ?        Sl     1:12 /opt/cloudera/parcels/CDH/lib/kudu/sbin/kudu-tserver

--rpc_authentication=required --rpc_encryption=required --keytab_file=/var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/kudu.keytab --tserver_master_addrs=master.myhost.org --flagfile=/var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/gflagfile

[root@anonymous ~]# klist -k /var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/kudu.keytab

Keytab name: FILE:/var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/kudu.keytab

KVNO Principal

---- --------------------------------------------------------------------------

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

If tablet servers are not runing or running without keytabs, or there is nothing in those keytabs, that might be the problem.

Anyway, I think there should be log files of Kudu tablet servers at those machines, by default they are in /var/log/kudu.  Checking those logs might give you some ideas what to start the troubleshooting with.

Regards,

Alexey

[#3]

There is documentation for how to enable Kudu security on CDH 5.13.0 here: https://www.cloudera.com/documentation/enterprise/5-13-x/topics/kudu_security.html#concept_syg_k35_l...

Please follow those steps and let us know if it still doesn't work for you.

Thanks,

Mike

[#4]

Hi Mike

It's this document that I followed to enable TLS/SSL and Kerberos.

I have this settings in Cloudera Manager

Is there something wrong ?

Best regards

Christophe

[#5]

Hi Alexey, Tablet Server are runnig, I've got the following : [993][root@XXX1111:~]# ps axw | grep kudu-tserver 61712 ? Sl 621:42 /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/kudu/sbin/kudu-tserver --rpc_authentication=required --rpc_encryption=required --keytab_file=/run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/kudu.keytab --tserver_master_addrs=XXX1105.krj.gie --flagfile=/run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/gflagfile 185450 pts/0 S+ 0:00 grep --color=auto kudu-tserver [994][root@knlXXX1:~]# klist -k /run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/kudu.keytab Keytab name: FILE:/run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/kudu.keytab KVNO Principal ---- -------------------------------------------------------------------------- 1 kudu/XXX1111.krj.gie@XXXX.GIE FYI I've got 5 tablets Server Is there anything wrong ? Regards Christophe

[#6]

The tablet servers are failing to register with the master. There should be errors in the tablet server and master logs about this, assuming that the value you've obfuscated in '--tserver_master_addrs=XXX1105.krj.gie' is correct for the Kudu master.

What do those errors say?

[#7]

The 'ps' sample output from one your servers looks fine.

Just another question: I assume the 'superuser_acl' property in you CM configuration (that's blurred out) contains 'kudu' (or whatever you have for the Kudu service principal), right?  If not, add that into the list.

Anyway, it's hard to say what's wrong looking at the configuration snippets and playing the 'guess what?' game.  I would highly recommend following Will's advise on looking into the logs of master(s) and tablet servers for the error details.  I think that will give you a firm starting point in troubleshooting the issue and save some time for everybody.

Regards,

Alexey

[#8]

Hi, Sorry for the late response, I was on another subject For your information : The property did not contain "kudu", I added it and it does not work. I have the same error when the property is set on "*" I've modified the kudu log level to WARNING and I have more messages On tablet server, I have this W1016 17:20:42.294350 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.308531 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.321985 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.339105 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.352322 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwbmxqylry', principal='kudu/XXX1115.krj.gie@REALM'} at 100.54.44.231:33470 W1016 17:20:42.373529 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.384279 48461 leader_election.cc:277] T eee6075900bf49a78a911fe5b88e98e3 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.384328 48461 leader_election.cc:277] T eee6075900bf49a78a911fe5b88e98e3 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.388542 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.393936 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.396759 48461 leader_election.cc:277] T db6bdb50bb964c40b76e73f633f82bc8 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.396842 48459 leader_election.cc:277] T db6bdb50bb964c40b76e73f633f82bc8 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.401067 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.402855 48459 leader_election.cc:277] T 7fbc0fdb102a4246937c14768eed58f1 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.402894 48461 leader_election.cc:277] T 7fbc0fdb102a4246937c14768eed58f1 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.408056 48461 leader_election.cc:277] T de0a76fa35154cfcba1d1acabbd9fd97 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.408090 48459 leader_election.cc:277] T de0a76fa35154cfcba1d1acabbd9fd97 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.418889 48461 leader_election.cc:277] T 789096c226c241b3bda456da99025455 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.418917 48459 leader_election.cc:277] T 789096c226c241b3bda456da99025455 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.421723 48461 leader_election.cc:277] T abc59c8abe454ff1b1e34f1b90f52851 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.421763 48459 leader_election.cc:277] T abc59c8abe454ff1b1e34f1b90f52851 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.459609 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.461763 48459 leader_election.cc:277] T 42e7bdc8c26642059439897f7f127f92 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.461799 48461 leader_election.cc:277] T 42e7bdc8c26642059439897f7f127f92 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.477473 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.488016 48459 leader_election.cc:277] T 7d80d9b648aa43c5b0e37e2aa882738b P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.488055 48461 leader_election.cc:277] T 7d80d9b648aa43c5b0e37e2aa882738b P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.525362 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.528292 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.553570 48459 leader_election.cc:277] T 358989d89bd541b88f0ebb78d47e650a P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.553606 48459 leader_election.cc:277] T 358989d89bd541b88f0ebb78d47e650a P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.588174 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.591974 48459 leader_election.cc:277] T 61f2c9b394f74eb49499ef1671d03596 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.592010 48459 leader_election.cc:277] T 61f2c9b394f74eb49499ef1671d03596 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.596632 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.603999 48461 leader_election.cc:277] T 9ae1a83fa31d4af6816d52e3351cf57e P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.604034 48461 leader_election.cc:277] T 9ae1a83fa31d4af6816d52e3351cf57e P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote I Can send the full log if you want. Thanks for your help Best Regards Christophe

[#9]

Hi, Sorry for the late response, I was on another subject

For your information : The property did not contain "kudu", I added it and it does not work. I have the same error when the property is set on "*"

I've modified the kudu log level to WARNING and I have more messages On tablet server, I have this :

W1016 17:20:42.294350 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.308531 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.321985 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.339105 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.352322 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwbmxqylry', principal='kudu/XXX1115.krj.gie@REALM'} at 100.54.44.231:33470 W1016 17:20:42.373529 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.384279 48461 leader_election.cc:277] T eee6075900bf49a78a911fe5b88e98e3 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.384328 48461 leader_election.cc:277] T eee6075900bf49a78a911fe5b88e98e3 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.388542 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.393936 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.396759 48461 leader_election.cc:277] T db6bdb50bb964c40b76e73f633f82bc8 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.396842 48459 leader_election.cc:277] T db6bdb50bb964c40b76e73f633f82bc8 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.401067 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.402855 48459 leader_election.cc:277] T 7fbc0fdb102a4246937c14768eed58f1 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.402894 48461 leader_election.cc:277] T 7fbc0fdb102a4246937c14768eed58f1 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.408056 48461 leader_election.cc:277] T de0a76fa35154cfcba1d1acabbd9fd97 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.408090 48459 leader_election.cc:277] T de0a76fa35154cfcba1d1acabbd9fd97 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.418889 48461 leader_election.cc:277] T 789096c226c241b3bda456da99025455 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.418917 48459 leader_election.cc:277] T 789096c226c241b3bda456da99025455 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.421723 48461 leader_election.cc:277] T abc59c8abe454ff1b1e34f1b90f52851 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.421763 48459 leader_election.cc:277] T abc59c8abe454ff1b1e34f1b90f52851 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.459609 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.461763 48459 leader_election.cc:277] T 42e7bdc8c26642059439897f7f127f92 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.461799 48461 leader_election.cc:277] T 42e7bdc8c26642059439897f7f127f92 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.477473 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.488016 48459 leader_election.cc:277] T 7d80d9b648aa43c5b0e37e2aa882738b P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.488055 48461 leader_election.cc:277] T 7d80d9b648aa43c5b0e37e2aa882738b P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.525362 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.528292 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.553570 48459 leader_election.cc:277] T 358989d89bd541b88f0ebb78d47e650a P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.553606 48459 leader_election.cc:277] T 358989d89bd541b88f0ebb78d47e650a P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.588174 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.591974 48459 leader_election.cc:277] T 61f2c9b394f74eb49499ef1671d03596 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.592010 48459 leader_election.cc:277] T 61f2c9b394f74eb49499ef1671d03596 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.596632 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.603999 48461 leader_election.cc:277] T 9ae1a83fa31d4af6816d52e3351cf57e P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.604034 48461 leader_election.cc:277] T 9ae1a83fa31d4af6816d52e3351cf57e P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote

I Can send the full log if you want.

Thanks for your help

Best Regards

Christophe

[#10]

Hi, Sorry for the late response, I was on another subject

[#11]

For your information : The property did not contain "kudu", I added it and it does not work. I have the same error when the property is set on "*"

[#12]

I've modified the kudu log level to WARNING and I have more messages On tablet server, I have this :

[#13]

W1016 17:20:42.294350 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.308531 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.321985 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.339105 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.352322 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwbmxqylry', principal='kudu/XXX1115.krj.gie@REALM'} at 100.54.44.231:33470 W1016 17:20:42.373529 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.384279 48461 leader_election.cc:277] T eee6075900bf49a78a911fe5b88e98e3 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.384328 48461 leader_election.cc:277] T eee6075900bf49a78a911fe5b88e98e3 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.388542 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.393936 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.396759 48461 leader_election.cc:277] T db6bdb50bb964c40b76e73f633f82bc8 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.396842 48459 leader_election.cc:277] T db6bdb50bb964c40b76e73f633f82bc8 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.401067 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.402855 48459 leader_election.cc:277] T 7fbc0fdb102a4246937c14768eed58f1 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.402894 48461 leader_election.cc:277] T 7fbc0fdb102a4246937c14768eed58f1 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.408056 48461 leader_election.cc:277] T de0a76fa35154cfcba1d1acabbd9fd97 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.408090 48459 leader_election.cc:277] T de0a76fa35154cfcba1d1acabbd9fd97 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.418889 48461 leader_election.cc:277] T 789096c226c241b3bda456da99025455 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.418917 48459 leader_election.cc:277] T 789096c226c241b3bda456da99025455 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.421723 48461 leader_election.cc:277] T abc59c8abe454ff1b1e34f1b90f52851 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.421763 48459 leader_election.cc:277] T abc59c8abe454ff1b1e34f1b90f52851 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.459609 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.461763 48459 leader_election.cc:277] T 42e7bdc8c26642059439897f7f127f92 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.461799 48461 leader_election.cc:277] T 42e7bdc8c26642059439897f7f127f92 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.477473 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.488016 48459 leader_election.cc:277] T 7d80d9b648aa43c5b0e37e2aa882738b P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.488055 48461 leader_election.cc:277] T 7d80d9b648aa43c5b0e37e2aa882738b P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.525362 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.528292 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-ibzddzigqg', principal='kudu/XXX1117.krj.gie@REALM'} at 100.54.44.233:41820 W1016 17:20:42.553570 48459 leader_election.cc:277] T 358989d89bd541b88f0ebb78d47e650a P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.553606 48459 leader_election.cc:277] T 358989d89bd541b88f0ebb78d47e650a P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.588174 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-hwefzjneur', principal='kudu/XXX1119.krj.gie@REALM'} at 100.54.44.235:59322 W1016 17:20:42.591974 48459 leader_election.cc:277] T 61f2c9b394f74eb49499ef1671d03596 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 041b5a3e1438484fbf3a68b10d91a928: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.592010 48459 leader_election.cc:277] T 61f2c9b394f74eb49499ef1671d03596 P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 36386227a1624b74895dd1fb6b3150e9: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.596632 48572 server_base.cc:329] Unauthorized access attempt to method kudu.consensus.ConsensusService.RequestConsensusVote from {username='m-zhdp-s-huzuudvhxu', principal='kudu/XXX1113.krj.gie@REALM'} at 100.54.44.229:42450 W1016 17:20:42.603999 48461 leader_election.cc:277] T 9ae1a83fa31d4af6816d52e3351cf57e P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer ea71c709fef34e0d87adfe90f917abc8: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote W1016 17:20:42.604034 48461 leader_election.cc:277] T 9ae1a83fa31d4af6816d52e3351cf57e P 3920550eeade417885b846064ddd2410 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer fa697a7fc04d4c62ae031c77db71be9b: Remote error: Not authorized: unauthorized access to method: RequestConsensusVote

[#14]

I Can send the full log if you want.

[#15]

Thanks for your help

[#16]

Best Regards

[#17]

Christophe

[#18]

Hi Christophe,

It seems in your case kudu service principals (like 'kudu/XXX1119.krj.gie@REALM') are not mapped into 'kudu' as expected, but into name of local users (like 'm-zhdp-s-hwefzjneur').  If I'm not mistaken, that's exactly https://issues.apache.org/jira/browse/KUDU-2198. 

As a workaround, I can suggest to add --use_system_auth_to_local=false to the Kudu flags (both masters and tservers).  If using CM, add that flag into the 'Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile'.

Hope this helps.

Regards,

Alexey

[#19]

Oh, sorry -- it seems you are at 5.13.0 and that flag is not available in that version yet (but it's present starting 5.14.0).  I'm afraid you need either to introduce custom mappings for those kudu service principals (so they would be mapped into 'kudu') or upgrade to 5.14 or higher to get access to that flag.  Setting superuser ACL to '*' would not allow tablet servers to register with masters anyway because of the following:

  https://github.com/apache/kudu/blob/master/src/kudu/master/master_service.cc#L122