To followup...   Got this working today.  Turns out it was caused by this setting...   hadoop.security.group.mapping=org.apache.hadoop.security.ShellBasedUnixGroupsMapping   Apparently this does a...   bash -c groups   ...for the user... which separates the groups by spaces.  When I changed to this implementation...   hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback   ... everything worked correctly.  Now hbase shell correctly lists the groups (even ones with spaces) and visibility labels works correctly.   That was a fun one... NOT!   Richard ... View more

One thing I noticed today in case it may help with this issue...   Today I tried the sqoop from MSQL -> Hbase again on a new table with compression set and pre-split in Cloudera 5.15.1 and Cloudera 6.2.1 environments,  Hbase configuration (and HDFS configuration for that matter) is almost identical.   In the Cloudera 6.2.1 (ie. Hbase 2.1.2) environment I see the flush to the HStoreFile happen fairly quickly (only about 32,000 new entries) and in the logs it mentions 'Adding a new HDFS file' of size 321Kb.   In the Cloudera 5.15.1 (ie. Hbase 1.2.x) environment I see the flush happen to the HStoreFile take longer and there are 700,000 entries being flush and the 'Adding a new HDFS file' is of size 6.5Mb.   The memstore flush size is set to 128Mb in both environments and region servers have 24Gb available.  So I think it's hitting the 0.4 heap factor for memstores and then it flushes in both cases.  Also there are only a few tables with heavy writes so most of the other tables are fairly idle.  So I don't think they would take up much memstore space.   In the Cloudera 6.2.1 environment each server holds about 240 regions.  In the Cloudera 5.15.1 environment each server holds about 120 regions.   My thinking is that if I can get the Cloudera 6.2.1/hbase 2.1.2 memstore flush happening with a similar size and number of entries as the Cloudera 5.15.1 environment the performance issue for large writes would be solved.   Just not sure how to make that happen.   I also noticed that minor compactions happen in both environments take a similar amount of time so I think that's not an issue.   Richard   ... View more

Hi Suku: I response some of your questions: a) Which Keytab you have used, whether CM generated keytab or user keytab generated by you? I used kafka.keytab b) Path of your jaas.conf and keytab for Kafka? Path of kafka.keytab in /etc/security/keytabs/ c) How Kafka Kerberos configuration parameters set? The following is the configuration of Kafka parameters and the the form to use the jaas parameter. Properties props = new Properties(); props.put("bootstrap.servers", "xxxx:9092,xxx:9092"); props.put("client.id", "client-id-coprocessor "); props.put("key.serializer", StringSerializer.class.getName()); props.put("value.serializer", StringSerializer.class.getName()); props.put("security.protocol", "SASL_PLAINTEXT"); props.put("sasl.kerberos.service.name", "kafka"); props.put("sasl.jaas.config", "com.sun.security.auth.module.Krb5LoginModule required \n" + "useKeyTab=true \n" + "storeKey=true \n" + "keyTab=\"/etc/security/keytabs/kafka.keytab\" \n" + "principal=\"kafka/nodo@REALM\";"); KafkaProducer producer = new KafkaProducerString>(props); Remember sometimes you will need reboot your hbase service for deploy your coprocessor. I hope I will help you. Florentino ... View more