Java code: DataFrame peopleDataFrame = sqlContext.createDataFrame(rowRDD, schema); HiveContext hiveContext = new org.apache.spark.sql.hive.HiveContext( jsc.sc() ); hiveContext.sql("CREATE TABLE IF NOT EXISTS people_t1 (emp_id string, first_name string, last_name string, job_title string, mgr_emp_id string)"); // Register the DataFrame as a table. peopleDataFrame.registerTempTable("people"); .... peopleDataFrame.insertInto("default.people_t1", true); Got: java.lang.RuntimeException: Table Not Found: default.people_t1 While table does exist in Hive: hive> describe people_t1; OK emp_id string first_name string last_name string job_title string mgr_emp_id string Time taken: 0.284 seconds, Fetched: 5 row(s) ... View more

Using our sandbox to load fixed-width file using pig: https://martin.atlassian.net/wiki/pages/viewpage.action?pageId=21299205 dump or store into works fine; however, when running with FixedWidthStorer, the job hanging--- no error: store employees into '/user/root/emps-regenerated' USING org.apache.pig.piggybank.storage.FixedWidthStorer('-10, 11-30, 31-50, 51-70, 71-','SKIP_HEADER'); 2015-12-02 18:35:39,386 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1449078859822_0001/ 2015-12-02 18:35:39,387 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1449078859822_0001 2015-12-02 18:35:39,387 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases employees 2015-12-02 18:35:39,387 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: employees[1,12] C: R: 2015-12-02 18:35:39,400 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2015-12-02 18:35:39,400 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1449078859822_0001] ... View more

Abbott has a third party tool, QlikView, which needs to run Hive queries. Suppose our cluster is kerberized via existing AD. Once HDP cluster is kerberized, QlikView needs to be kerberized as well----it can get through Knox, but let's take Knox out of the picture for this discussion. My questions are: 1. since both QlikView and HDP cluster are kerberized via the same AD, QlikView ticket should be recognized by HDP cluster. But, is there any extra setting needed? do we need to put QlikView service Account to OU=HadoopService? 2. QlikView is a server, user can log in to QlikView through AD. I believe Hive ACLs applies to each individual user no matter they are via QlikView or not --- this question is more like to QlikView guy, not us. ... View more

HDP 2.2.8, Ambari 2.1.2, non-kerberoized cluster. Client is trying to turn on capacity queue ACL--- only certain users are allowed to submit jobs to a queue; but unlisted users can still submit queues after "save and refresh queues". Also, got this error wheile refreshServiceAcl [nveyarn@hg003rdv ~]$ yarn rmadmin -refreshServiceAcl refreshServiceAcl: java.io.IOException: Service Authorization (hadoop.security.authorization) not enabled. ... View more

Client tries to execute hive sql inside a file via Beeline. Anyone has working sample of how to use "beeline -u <url> -n <user> -f <file> -p <password>"? I have tried this, but does not work. ... View more