Hello,   I have configured CDH cluster with LDAP integration and CompositeGroupMapping (ShellBasedUnixGroupsMapping and LdapGroupsMapping) on HDFS. HDFS, Hive and Impala works great with both local user principals as well as AD users.   The problem I have now is with Spark (on YARN), where jobs submitted by local users work, but those submitted by AD users fail:   main : run as user is ldap1 main : requested yarn user is ldap1 User ldap1 not found   If I create user ldap1 on all hosts, then Spark works. What am I missing here?   Thank you ... View more

Hello,   I noticed the following behaviour:   1. I have set the following:   <property> <name>hive.metastore.kerberos.keytab.file</name> <value>/etc/hive/conf/hive.keytab</value> </property> 2. I have generated and placed the hive.keytab in both MetaStoreServer and HS2 nodes   Now, each time I restart Hive these files are deleted as the whole /etc/hive/conf directory is re-created.   Am I doing something wrong? ... View more

Hello,   I am managing a CDH 5.13 cluster with 4 datanodes. Each datanode had 10 x 2.7 TB disks (~90% used) and we just added another 8 x 3.6 TB disks on each node.   I did a "Rebalance" on HDFS service, which apparently did nothing as all nodes have the same disk usage (in total).   Now, I followed this post in order to intra-node-balance the disks (with threshold set to 25). After 1 hour of execution, the progress is terribly slow (as you can see in the last /data/18 node which gets data):     $ sudo df -h ... /dev/sdc1 2.8T 2.4T 361G 88% /data/02 /dev/sdk1 2.8T 2.5T 315G 89% /data/10 /dev/sdg1 2.8T 2.5T 308G 89% /data/07 /dev/sdi1 2.8T 2.5T 314G 89% /data/08 /dev/sdj1 2.8T 2.5T 300G 90% /data/09 /dev/sde1 2.8T 2.5T 299G 90% /data/04 /dev/sdf1 2.8T 2.5T 303G 90% /data/06 /dev/sdh1 2.8T 2.4T 353G 88% /data/05 /dev/sdb1 2.8T 806G 2.0T 29% /data/01 /dev/sdd1 2.8T 2.5T 298G 90% /data/03 ---#NEW DISKS#--- /dev/sdl1 3.7T 35M 3.7T 1% /data/11 /dev/sdm1 3.7T 36M 3.7T 1% /data/12 /dev/sdn1 3.7T 34M 3.7T 1% /data/13 /dev/sdo1 3.7T 35M 3.7T 1% /data/14 /dev/sdp1 3.7T 34M 3.7T 1% /data/15 /dev/sdq1 3.7T 34M 3.7T 1% /data/16 /dev/sdr1 3.7T 34M 3.7T 1% /data/17 /dev/sds1 3.7T 26G 3.7T 1% /data/18     I would like to ask the following:    1. Currently, there are not pipelines accessing HDFS, but tomorrow morning there will be, and it's obvious from the progress that disk balancing won't have finished. Is it safe to leave this process to finish, while having the cluster in production?     2. Is there something I can do to speed things up?    3. How can I terminate this process, safely, if required?   Thank you, Gerasimos ... View more

Hello,   I have a working (for some months) Java UDF for Impala. The CDH cluster I am working has 4 Impala Daemons. Today, this UDF behaved very strange after a DROP & CREATE:   1. From HUE (Impala) the CREATE FUNCTION succeeded and SHOW FUNCTIONS reported the right signature. 2. Then, from impala-shell I tried call the function but I got an ImpalaRuntimeException:   select phoneme_index("test","test","test","test",true); ImpalaRuntimeException: Unable to find evaluate function with the correct signature: my.UDF.PhonemesIndex.evaluate(STRING, STRING, STRING, STRING, BOOLEAN) UDF contains:     public java.lang.Double my.UDF.PhonemesIndex.evaluate(java.lang.String,java.lang.String,java.lang.String,java.lang.String)   3. Then, I logged in Hue and tested the same SQL query, which returned "1" as expected.   After several tries, I finally figured that if I connect with impala-shell -i to a different daemon, the function works as expected!   Any ideas why is this happening? It is the first time I am coming across with such an issue. ... View more