Context: HDP 3.1 cluster.  Issue: After specifying the value for `` hive.server2.tez.interactive.queue`` in Custom hiveserver2-interactive-site, we noticed that in the Yarn Application Manager there are tons of new TEZ instances that are under the same specified interactive queue.    The issue is that the number of these instances is too large for our needs. How can we control the number of TEZ instances for  hive.server2.tez.interactive.queue?  ... View more

Running a HDP 3.1 cluster with Hive LLAP enabled. The main problem was working with many concurrent Hive queries. Some of them were very large, occupying the whole cluster, making other concurrent queries waiting for the first one to finish.   For this reason I've followed these steps from documentation. 1. Created two resource plans for slow queries and fast queries, based on: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/hive-workload/content/hive_setting_up_and_using_a_resource_plan.html  > show resource plans ; +------------------+---------+--------------------+ | rp_name | status | query_parallelism | +------------------+---------+--------------------+ | high_concurr_rp | ACTIVE | NULL | +------------------+---------+--------------------+   2. Configured the workflow management, based on this: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/hive-workload/content/hive_interactive_queue.html I now have `` hive.server2.tez.interactive.queue=interactive-q`` in custom-site.    My questions are these: 1. How can I check whether those rules from the examples are working? Can I somehow see which pool and what resource allocation is a specific query getting? 2. With the addition of `` hive.server2.tez.interactive.queue``, I've noticed that LLAP is spawning tons of TEZ applications in the ``interactive-q`` queue, so much so that many are not fully running (state=ACCEPTED, FinalStatus=UNDEFINED). How can I control the number (or disable this)? In the ``llap`` queue we have 1 ``llap0`` running as a yarn-service, with 3 supporting TEZ applications.    ... View more

I'm trying to monitor the job status of a running Oozie job. However, running the following code from a general purpose user ("hue", which is also starting these jobs through runAs):     curl --negotiate -u : http://FQDN:8088/proxy/application_1609929757167_0004/ws/v1/mapreduce/jobs/job_1609929757167_0004/tasks     I'm getting an error, which does not really say anything useful:   {"RemoteException":{"exception":"WebApplicationException","javaClassName":"javax.ws.rs.WebApplicationException"}}     On the other hand, I'm able to see the correct response through a yarn user, suggesting some issues with the ACL's.   My question: what ACL's and where should I give to the general purpose user so that it could see all the information of the running task? I've added the user to ``yarn.admin.acl``, as well as  mapreduce.job.acl-modify-job, mapreduce.job.acl-view-job, mapreduce.jobhistory.admin.acl, but none of these help.   What am I missing?  ... View more

I'm running HDP 3.1 cluster with a TEZ-UI instance configured separately on Apache Tomcat. Configured everything following this guide. Everything seems to be working fine, however, when launching Hive LLAP queries I'm not seeing generated DAG IDs, and, for example, can't see the error messages after the query crashed.    Example screenshot: What additional configuration is necessary for this to work?    Sidenotes: Don't know whether this matters, but in the same manner the LLAP Master monitor (:10502 port) does not register any queries, even though Hive is configured to use LLAP and executors are always active and called (when inspecting nodes at :15002 port), e.g.: Meanwhile in :15002, LLAP workers are used actively:     ... View more

I'm trying to upgrade the Kerberos encryption types for an existing HDP 2.6 cluster. The problem is that I want to use the same KDC servers with a single configuration for multiple realms, where one would be set up with a Centos 7.8 or Centos 8, which does not support DES-type encryptions.    The HDP2.6 cluster does not seem to work with other than the DES encryptions.    I'm trying the following krb5.conf:     default_tkt_enctypes = aes256-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-cbc-md4 default_tgs_enctypes = aes256-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-cbc-md4     In my understanding, these should cover both the old and weak des3-cbc-sha1 types, and the later aes256 types for the newer system.    However, with this configuration set and doing the Keytab Regeneration through Ambari, the HDFS services doesn't start due to a probably GSS issue (same errors as in https://community.cloudera.com/t5/Support-Questions/Cloudera-Kerberos-GSS-initiate-failed/m-p/78727). When inspecting the auto-generated keytab, only one entry is created with "des3-cbc-sha1" tab. While this should work (and it does allow for a kinit), something is not okay for the namenode and it still results in the GSS errors while starting the namenode.    What could be the issue here?    What is the correct setting for kerberos enctypes that works with HDP?   -------- I can reform the question in the following manner:   Why does HDFS Namenode works (on HDP 2.6) only with the following krb5.conf entries:   default_tkt_enctypes = des3-cbc-sha1 des3-hmac-sha1 des3-cbc-sha1-kd default_tgs_enctypes = des3-cbc-sha1 des3-hmac-sha1 des3-cbc-sha1-kd permitted_enctypes = des3-cbc-sha1 des3-hmac-sha1 des3-cbc-sha1-kd   Nothing else works if I try moving away from the DES encryption.  ... View more

Hi all,   what is the current best practice for setting up a HDP 3.1 framework on a single machine of 20 cores and 512GB RAM?    Do we still need to split up the machine to, e.g., 2 x VM, in order to ensure at least 2x dfs replication? Or would a single node cluster be more efficient?    If I understand correctly, even if using a JBOD disk setup for the HDFS, with a single node it cluster it will be possible to use at most 1x replication through the ``dfs.replication`` setting?    What are your recommendations at this point?    @Shelton  ... View more

Running on HDP2.6. This query seemed to be working fine (a simple create table operation; just doing a select works without probems), but now Hive LLAP returns the following error when trying to create a table: Error running query: java.lang.AssertionError: Unexpected type UNEXPECTED from: 2019-08-06T17:29:20,701 WARN  [HiveServer2-Handler-Pool: Thread-1858]: thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(516)) - Error executing statement: org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.AssertionError: Unexpected type UNEXPECTED         at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:225) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:276) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.cli.operation.Operation.run(Operation.java:312) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:508) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:495) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:309) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:599) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Caused by: java.lang.AssertionError: Unexpected type UNEXPECTED         at org.apache.hadoop.hive.ql.parse.CalcitePlanner.fixUpCtasAndInsertAfterCbo(CalcitePlanner.java:952) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:378) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11167) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:290) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:257) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:455) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1197) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1184) ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:191) ~[hive-service-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]         ... 15 more This is the first time I'm seeing such a error. What does it even link to? Any known solutions? EDIT: Important comment: this is 100% related to masking through Apache Ranger of some columns in the used tables. However, both the imposed masking AND the ability to create new tables using the initial masked one is important. Any known workaround? ... View more

Hi all, After setting up a fresh kerberized HDP 3.1 cluster with Hive LLAP, Spark2 and Livy, we're having trouble connecting to Hive's database through Livy. Pyspark from shell works without the problem, but something breaks when using Livy. 1. Livy settings are Ambari default, with additionally specified jars and pyfiles for the HWC connector, spark.sql.hive.hiveserver2.jdbc.url and spark.security.credentials.hiveserver2.enabled true. These are enough for pyspark shell to work without problems. 2. Connection is made through the latest HWC connector described here, since apparantly this is the only one that works for Hive 3 and Spark2. problem: 1. When spark.master is set to yarn client mode (See for example the comment here), the connector appends a principal "hive/_HOST@DOMAIN" and the connection returns GSS error - failing to find any Kerberos tgt (although, the ticket is there and livy has access to the hiveserver2). 2. When spark.master is set to yarn cluster mode, ";auth=delegationToken" is appended to the connection, where the error follows that "PLAIN" connection is made, where a kerberized one is expected. Notes: tried various settings -- zookeeper jdbc links vs direct through port 10500, hive.doAs = true vs false, various principals, but nothing works. Note2: everything works fine when connecting both through beeline (to hive at 10500 port) and through pyspark shell. Note3: HWC Connection snippet (from examples): from pyspark_llap import HiveWarehouseSession hive = HiveWarehouseSession.session(spark).build() hive.showDatabases().show(100) Any ideas? Feel like some setting on Livy is missing, especially weird seeing that "failed to find any Kerberos tgt" - where is it looking for it and why doesn't it see the ticker from "kinit"? @Geoffrey Shelton Okot @Hyukjin Kwon @Eric Wohlstadter ... View more

Hello all, I have set up a HDP 3 cluster with Hive running on LLAP-Interactive. However, when connecting through Hiveserver2 port (10500 default) (or if through zookeeper, then through Hiveserver2-Interactve), any query that is being run is going through YARN/TEZ containers. That is, if I check at the LLAP Dashboard (http://datanode:10502), I can see that there are 0 active/passive sessions. With my experience from HDP2.6, there should be every query listed. What can be the problem? LLAP is enabled with the default settings from Ambari 2.7, with the following settings: hive.execution.mode = llap #at hive-interactive-site hive.llap.execution.mode = all #same with the default "only" hive.execution.mode = container #at hive-site Is this the default behaviour? Currently I'm worried that there are two different "hive.execution.mode" settings specified, so will try playing with them. But other than that, I'm not sure what further settings to check. @Geoffrey Shelton Okot ... View more

Hello all, after fresh kerberization of Ambari 2.7.3 / HDP 3 cluster, the HDFS namenode isn't able to start because the hdfs user can't talk to the webhdfs. The following error is returned: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) It is not only from ambari: I can recreate this error from a simple curl call from hdfs user: su - hdfs curl --negotiate -u : http://datanode:50070/webhdfs/v1/tmp?op=GETFILESTATUS Which returns </head> <body><h2>HTTP ERROR 403</h2> <p>Problem accessing /webhdfs/v1/tmp. Reason: <pre> GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)</pre></p> </body> </html> Overall permission for this user should be in tact, since I'm able to run hdfs operations from shell and kinit without problems. What could be the problem? I've tried recreating keytabs several times, and fiddling with ACL settings on the config, but nothing works. What principal is WEBHDFS expecting? The same results are when I'm trying accessing it with HTTP/host@EXAMPLE.COM principal. NB: I'll add that there's nothing fancy in the HDFS settings, mainly stock/default config. NB2: I will add, that I've added all possible encryption types to krb5.conf as I could find, but none if these helped: default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal permitted_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal @Geoffrey Shelton Okot ... View more

We will be setting up a HDP 3 cluster, and the idea is to split 1xRAID1 and 2xHDD through 2 virtual machines. 1xRAID1 would handle the OS and local user files, while each of the 2xHDD's would be used for hdfs /grid/. My question: is 2xHDD enough to ensure file redundancy? I've read that Hadoop 3 does not require 3 replicas as it did in Hadoop 2, however, in case of a drive failure, would it still ensure that the files on the remaining HDD are enough for a full recovery? Or would you advice partitioning some of the 1xRAID1 for a third /grid/ drive for the hdfs? In this case it seems to introduce a lot of overhead due to RAID. What is the minimum amount of HDD's required for redundant hdfs operation? ... View more

We will be setting up a HDP 3 cluster, and the idea is to split 1xRAID1 and 2xHDD through 2 virtual machines. 1xRAID1 would handle the OS and local user files, while each of the 2xHDD's would be used for hdfs /grid/. My question: is 2xHDD enough to ensure file redundancy? I've read that Hadoop 3 does not require 3 replicas as it did in Hadoop 2, however, in case of a drive failure, would it still ensure that the files on the remaining HDD are enough for a full recovery? Or would you advice partitioning some of the 1xRAID1 for a third /grid/ drive for the hdfs? In this case it seems to introduce a lot of overhead due to RAID. What is the minimum amount of HDD's required for redundant hdfs operation? Big thanks! ... View more

Half of the resources on YARN in our cluster is dedicated to Hive LLAP, while the other half is left free for other various tasks, such as Spark or mapreduce containers. However, there are times when no other jobs are running, and LLAP is loaded to its fullest. Is there a setting allowing for dynamic allocation of all resources? That is, in such cases, would it be possible to allocate all, or i.e., 80% of the unused space for LLAP, and return it back to normal after finishing the jobs? As far as I know now, only way to do that would require a LLAP or YARN restart, which is not at all preferred. ... View more