Reply
Explorer
Posts: 9
Registered: ‎08-02-2018

Re: Kudu and Kerberos

  1. When was the last time the cluster worke?
  2. What has changed since then?

 

I have this issue when I check this :

 

2018-09-17 10_50_25-Kudu - Cloudera Manager.png

Cloudera Employee
Posts: 17
Registered: ‎03-16-2017

Re: Kudu and Kerberos

Hi,

 

Can you check what keytab your tablet servers are running with?

 

You can do that by logging in to one of the tablet server machines and checking the command line that kudu-tserver process is running with.   Then check what's inside that keytab.

 

It's something like

 

[root@anonymous ~]# ps axw | grep kudu-tserver

  548 pts/0    S+     0:00 grep --color=auto kudu-tserver

32747 ?        Sl     1:12 /opt/cloudera/parcels/CDH/lib/kudu/sbin/kudu-tserver

--rpc_authentication=required --rpc_encryption=required --keytab_file=/var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/kudu.keytab --tserver_master_addrs=master.myhost.org --flagfile=/var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/gflagfile

 

[root@anonymous ~]# klist -k /var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/kudu.keytab

Keytab name: FILE:/var/run/cloudera-scm-agent/process/580-kudu-KUDU_TSERVER/kudu.keytab

KVNO Principal

---- --------------------------------------------------------------------------

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

   2 kudu/ts-01.myhost.org@DC.MYHOST.ORG

 

 

If tablet servers are not runing or running without keytabs, or there is nothing in those keytabs, that might be the problem.

 

Anyway, I think there should be log files of Kudu tablet servers at those machines, by default they are in /var/log/kudu.  Checking those logs might give you some ideas what to start the troubleshooting with.

 

 

Regards,

 

Alexey

Cloudera Employee
Posts: 64
Registered: ‎04-08-2014

Re: Kudu and Kerberos

There is documentation for how to enable Kudu security on CDH 5.13.0 here: https://www.cloudera.com/documentation/enterprise/5-13-x/topics/kudu_security.html#concept_syg_k35_l...

 

Please follow those steps and let us know if it still doesn't work for you.

 

Thanks,

Mike

Explorer
Posts: 9
Registered: ‎08-02-2018

Re: Kudu and Kerberos

 

Hi Mike

 

It's this document that I followed to enable TLS/SSL and Kerberos.

 

I have this settings in Cloudera Manager2018-09-25 10_18_30-Kudu - Cloudera Manager.png

 

2018-09-25 10_21_29-Security - Cloudera Manager.png

 

2018-09-25 10_23_13-Kudu - Cloudera Manager.png

2018-09-25 10_25_25-Kudu - Cloudera Manager.png

 

2018-09-25 10_26_02-Kudu - Cloudera Manager.png

 

Is there something wrong ?

Best regards

 

Christophe

 

 

Explorer
Posts: 9
Registered: ‎08-02-2018

Re: Kudu and Kerberos

Hi Alexey, Tablet Server are runnig, I've got the following : [993][root@XXX1111:~]# ps axw | grep kudu-tserver 61712 ? Sl 621:42 /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/kudu/sbin/kudu-tserver --rpc_authentication=required --rpc_encryption=required --keytab_file=/run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/kudu.keytab --tserver_master_addrs=XXX1105.krj.gie --flagfile=/run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/gflagfile 185450 pts/0 S+ 0:00 grep --color=auto kudu-tserver [994][root@knlXXX1:~]# klist -k /run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/kudu.keytab Keytab name: FILE:/run/cloudera-scm-agent/process/7996-kudu-KUDU_TSERVER/kudu.keytab KVNO Principal ---- -------------------------------------------------------------------------- 1 kudu/XXX1111.krj.gie@XXXX.GIE FYI I've got 5 tablets Server Is there anything wrong ? Regards Christophe
Cloudera Employee
Posts: 6
Registered: ‎04-25-2017

Re: Kudu and Kerberos

The tablet servers are failing to register with the master. There should be errors in the tablet server and master logs about this, assuming that the value you've obfuscated in '--tserver_master_addrs=XXX1105.krj.gie' is correct for the Kudu master.

What do those errors say?
Cloudera Employee
Posts: 17
Registered: ‎03-16-2017

Re: Kudu and Kerberos

The 'ps' sample output from one your servers looks fine.

 

Just another question: I assume the 'superuser_acl' property in you CM configuration (that's blurred out) contains 'kudu' (or whatever you have for the Kudu service principal), right?  If not, add that into the list.

 

Anyway, it's hard to say what's wrong looking at the configuration snippets and playing the 'guess what?' game.  I would highly recommend following Will's advise on looking into the logs of master(s) and tablet servers for the error details.  I think that will give you a firm starting point in troubleshooting the issue and save some time for everybody.

 

 

Regards,

 

Alexey

Announcements