@manjilhk Thanks for sharing this awesome solution using THttpClient transport, can you let us know which CDH version are you at? In CDH6.x the TSaslClientTransport is working, but in CDP starter version there's some code changed to cause this transport failed to communicate with secured cluster. We have released hotfix to this issue, if below KB matches your issue please raise a Cloudera case to apply for this hotfix, or you need to wait for the future release 7.2.11 which will include this fix. Please see this KB that I posted: https://my.cloudera.com/knowledge/Cannot-connect-to-HBase-Thrift-from-Python-scripts-after?id=318921 - Will Xiao, Support Engineer Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, click on the thumbs up button.
... View more
Update: It is found that .hiverc was used in hive user for HIVE CLI so the difference was found. hive.exec.scratchdir=/user/hive/scratch hive.exec.stagingdir=/user/hive/staging The issue is hdfs /user/hive directory is encrypted with Ranger and hdfs /tmp/hive directory is non-encrypted and can read/write by all user in hadoop group. hive-site.xml <property> <name>hive.security.authorization.sqlstd.confwhitelist.append</name> <value>hive\.exec\.scratchdir|hive\.exec\.stagingdir</value> <description>append conf property in white list followed by pipeline</description> </property> Restart the metastore and hiveserver. I tested with beeline with session level change . The execution is fast like HIVE CLI . hive.exec.scratchdir=/user/hive/scratch hive.exec.stagingdir=/user/hive/staging I tested with HIVE CLI with session level change. The execution is slow with MAP reduce Job for moving data. hive.exec.scratchdir=/tmp/hive/scratch hive.exec.stagingdir=/tmp/hive/staging So the root cause is data are encrypted in /user/hive and not encrypted in /tmp/hive. Solution is to make ssession level change to use same encryption zone. So below INFO log will be printed if the encryption zones are different. metadata.Hive: Copying source hdfs://edhcluster/tmp/hive/staging_hive_2018-08-07_16- 29-12_750_8973639287951385407-1/-ext-10000/000001_0 to hdfs://edhcluster/user/hive/warehouse/temp_tro/000001_0 because HDFS encryption zones are different. Thanks, Manjil
... View more