Support Questions
Find answers, ask questions, and share your expertise

how to set JSVC_HOME on secure data node

Hi Expert,

I am manual configuring kerberos on CDH 5.4, according to the guide  , it required to set below  on all data node in /etc/default/hadoop-hdfs-datanode: 

export HADOOP_SECURE_DN_USER=hdfs
export HADOOP_SECURE_DN_PID_DIR=/var/lib/hadoop-hdfs
export HADOOP_SECURE_DN_LOG_DIR=/var/log/hadoop-hdfs
export JSVC_HOME=/usr/lib/bigtop-utils/

And , Depending on the version of Linux you are using, you may not have the /usr/lib/bigtop-utils directory on your system. If that is the case, set the JSVC_HOME variable to the /usr/libexec/bigtop-utils directory by using this command.

 

My CDH 5.4 is managed by CM. My questioin is :

1, There is no file or directory  /etc/default/hadoop-hdfs-datanode; How do i change it , hadoop-evn.sh?

2, There is no  bigtop-utils installed on my data node, so, neither dierectory (/usr/lib/bigtop-utils/ or /usr/libexec/bigtop-utils ) exist.  Do I need to install it?

Thanks,

David 

 

9 REPLIES 9

Re: how to set JSVC_HOME on secure data node

Hi,

When using Cloudera Manager, you should follow Cloudera Manager instructions for configuring things like Kerberos, which has a wizard to assist you:
http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_sg_authenticatio...
http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_sg_intro_kerb.ht...

Cloudera Manager does not generally configure roles to read from /etc, so any instructions you find telling you to do that won't work. You should never directly modify hadoop-related files in /etc when using Cloudera Manager. Instead, you need to tell Cloudera Manager what configs you want, then Cloudera Manager can deploy those changes to the appropriate files across your cluster.
http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_intro_primer.htm...

Thanks,
Darren

Re: how to set JSVC_HOME on secure data node

Hi Darren,

Thanks a lot for your response.

Yes, I have configureing kerberos sucesfully on CDH 5.4 without isilon HDFS by CM wizzard. It works fine.

But now, I am configuring  kerberos on CDH 5.4 with isilon HDFS. The CM auto wizzard can not work. So, I have to manually enable/configure kerberos.

 

The "JSVC_HOME" configuration is according to "Cloudera Security Guide"  as below link

http://www.cloudera.com/content/www/en-us/documentation/enterprise/5-4-x/topics/cdh_sg_set_variables...

 

Could you help to check?

Thanks,

David 

 

 

Re: how to set JSVC_HOME on secure data node

Hi David,

Follow these steps to enable kerberos without the wizard. Since you're using Isilon, a few things will look a bit different, mostly where certain HDFS configs don't exist in Isilon, but it should largely work.
http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_sg_using_cm_sec_...

You should not need to set JSVC_HOME since there are no DataNodes when using Isilon.

Thanks,
Darren

Re: how to set JSVC_HOME on secure data node

Re: how to set JSVC_HOME on secure data node

Hi Darren,

Yes, seems the issue is not causedby JSVC.

I do a simple test to install bigtop-utils on one of the name node. And set environment variable as CDH security guide. I got the same error.

 

I will try the way to recommend (without  CM wizzard to enabled kerb).

 

One of my concern is , I may meet the same issue when using CM wizzard. As described below:

http://community.cloudera.com/t5/Cloudera-Manager-Installation/How-to-change-hdfs-site-xml/m-p/34855...

 

CM will generate   hdfs-site.xml automatically. And I can not change it according to isilon integration guide.

 

I am configuring Kerberos on CDH. I need to change hdfs-site.xml from

//////////

  <property>

    <name>dfs.namenode.kerberos.principal</name>

    <value>hdfs/_HOST@FBDLSEC.LOCAL</value>

  </property>

  <property>

    <name>dfs.datanode.kerberos.principal</name>

    <value>hdfs/_HOST@FBDLSEC.LOCAL</value>

  </property>

////////////

To

  <property>

    <name>dfs.namenode.kerberos.principal</name>

    <value>hdfs/vIsilon-sec@FBDLSEC.LOCAL</value>

  </property>

  <property>

    <name>dfs.datanode.kerberos.principal</name>

    <value>hdfs/vIsilon-sec@FBDLSEC.LOCAL</value>

  </property>

Thanks,

David 

Re: how to set JSVC_HOME on secure data node

Hi David,

CM handles updating hdfs-site.xml to integrate with Isilon, as long as you've configured CM to use Isilon.

One of the same pages I linked before talks about how to configure Isilon within CM.
http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_mc_isilon_servic...

Thanks,
Darren

Re: how to set JSVC_HOME on secure data node

Hi Darren,

Appreciate  for your great help.

I tried no-wizzard kerb config, it almost works! And I believe auto-wizzard can also work. 

But before using the wizzard, there are strict configuraiton requried on isilon.

Still have 2 small issue, seems it caused by kerb config(since there is no such issue before kerb). Could you kindly suggest?

1, HDFS 

Canary test failed to create parent directory for /tmp/.cloudera_health_monitoring_canary_files.

 

2, HIVE

The Hive Metastore canary failed to create the hue hdfs home directory.

 

Thanks,

David

Re: how to set JSVC_HOME on secure data node

Hi David,

You should not have an HDFS service. Unfortunately, the fact that it is present probably means everything is configured to use HDFS and it all needs to be cleaned up and re-initialized on Isilon, following the Isilon instructions I linked before.

Thanks,
Darren

Re: how to set JSVC_HOME on secure data node

Darren,

I will check that according the guide you provide.  Appreciate for your kindly help.

Since the CDH cluster works fine before I enable kerberos. Seems the issue may caused by some ACL , file permission configuratioin, impersenation config...

 

I took some test below, could you give me some hints(direction) based on your powerful experience&expertise skill :)

 

1, Canary test failed to create parent directory for /tmp/.cloudera_health_monitoring_canary_files

I do test as below, seems  account hdfs can  mkdir directory  in /tmp.  The /tmp permission configure seems OK. Why it can not create canary temp directory?

/////////////////////

[root@clou3cdhworker0 ~]# kinit hdfs@FBDLSEC.LOCAL -kt /tmp/hdfsroot.keytab

[root@clou3cdhworker0 ~]# klist

Ticket cache: FILE:/tmp/krb5cc_0

Default principal: hdfs@FBDLSEC.LOCAL

 

Valid starting     Expires            Service principal

12/10/15 01:44:54  12/11/15 01:44:54  krbtgt/FBDLSEC.LOCAL@FBDLSEC.LOCAL

        renew until 12/17/15 01:44:54

[root@clou3cdhworker0 ~]# hadoop fs -ls /

Found 5 items

-rw-r--r--   1 root  hadoop              0 2015-12-04 08:07 /THIS_IS_ISILON_cloudyBT_cloudyBT-zone

drwx------   - hbase hbase               0 2015-12-10 01:40 /hbase

drwxrwxr-x   - solr  solr                0 2015-12-04 08:47 /solr

drwxrwxrwt   - hdfs  supergroup          0 2015-12-10 01:45 /tmp

drwxr-xr-x   - hdfs  supergroup          0 2015-12-04 08:47 /user

[root@clou3cdhworker0 ~]# hadoop fs -mkdir /tmp/.12345

[root@clou3cdhworker0 ~]# hadoop fs -ls /tmp

Found 8 items

drwxr-xr-x   - root   supergroup          0 2015-12-10 01:45 /tmp/.12345

……

/////////////////////////////////////////

 

B, The Hive Metastore canary failed to create the hue hdfs home directory.

[root@clou3cdhworker0 ~]# hadoop fs -ls /user/

Found 8 items

drwx------   - hdfs   supergroup          0 2015-12-10 00:45 /user/hdfs

drwxrwxrwx   - mapred hadoop              0 2015-12-04 08:47 /user/history

drwxrwxr-t   - hive   hive                0 2015-12-04 08:47 /user/hive

drwxrwxr-x   - hue    hue                 0 2015-12-04 08:47 /user/hue

drwxrwxr-x   - impala impala              0 2015-12-04 08:47 /user/impala

drwxrwxr-x   - oozie  oozie               0 2015-12-04 08:47 /user/oozie

drwxr-x--x   - spark  spark               0 2015-12-04 08:47 /user/spark

drwxrwxr-x   - sqoop2 sqoop               0 2015-12-04 08:47 /user/sqoop2

///////////////////////////////////////////

 

C, When using beeline connect to Hive with user principal test01@FBDLSEC.LOCAL, I got error:

A, Hive permissioin for beeline

[root@clou3cdhworker0 ~]# kinit test01@FBDLSEC.LOCAL

Password for test01@FBDLSEC.LOCAL:

[root@clou3cdhworker0 ~]# beeline

Beeline version 1.1.0-cdh5.4.4 by Apache Hive

beeline> !connect jdbc:hive2://172.16.154.23:10000/default;principal=hive/clou3cdhmaster0.fbdlsec.local@FBDLSEC.LOCAL

scan complete in 2ms

Connecting to jdbc:hive2://172.16.154.23:10000/default;principal=hive/clou3cdhmaster0.fbdlsec.local@FBDLSEC.LOCAL

Enter username for jdbc:hive2://172.16.154.23:10000/default;principal=hive/clou3cdhmaster0.fbdlsec.local@FBDLSEC.LOCAL:

Enter password for jdbc:hive2://172.16.154.23:10000/default;principal=hive/clou3cdhmaster0.fbdlsec.local@FBDLSEC.LOCAL:

Error: Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: mkdirs(/tmp/hive/test01) failed: Permission denied (state=,code=0)

0: jdbc:hive2://172.16.154.23:10000/default (closed)>

 

View /tmp/hive permission, seems all user can change it.

[root@clou3cdhworker0 ~]# hadoop fs -ls /tmp

Found 7 items

drwxr-xr-x   - root   supergroup          0 2015-12-10 01:16 /tmp/.abc

drwxr-xr-x   - root   supergroup          0 2015-12-10 01:20 /tmp/123

drwxr-xr-x   - root   supergroup          0 2015-12-10 00:40 /tmp/abc

drwx--x--x   - hbase  supergroup          0 2015-12-09 16:01 /tmp/hbase-staging

drwx-wx-wx   - hive   supergroup          0 2015-12-04 08:47 /tmp/hive

drwxrwxrwt   - mapred hadoop              0 2015-12-04 08:49 /tmp/logs

drwxr-xr-x   - mapred supergroup          0 2015-12-04 08:47 /tmp/mapred

[root@clou3cdhworker0 ~]# hadoop fs -ls /tmp/hive

ls: Permission denied

[root@clou3cdhworker0 ~]# hadoop fs -mkdir /tmp/hive/test01

mkdir: mkdirs(/tmp/hive/test01) failed: Permission denied

[root@clou3cdhworker0 ~]# klist

Ticket cache: FILE:/tmp/krb5cc_0

Default principal: test01@FBDLSEC.LOCAL

Valid starting     Expires            Service principal

12/10/15 01:24:50  12/11/15 01:24:50  krbtgt/FBDLSEC.LOCAL@FBDLSEC.LOCAL

        renew until 12/17/15 01:24:50

Thanks,

David