About shigeru_takehar

shigeru_takehar · ‎12-08-2016

The solution was: The Spark provide a sample HBase test program in /usr/hdp/current/spark-client/examples/src/main/scala/org/apache/spark/examples. The program name is HBaseTest.scala. If you open this file, you will see the comment: // please ensure HBASE_CONF_DIR is on classpath of spark driver // e.g: set it through spark.driver.extraClassPath property // in spark-defaults.conf or through --driver-class-path // command line option of spark-submit So, I added that parameter and my command line becomes as follows: spark-submit --jars hive-hbase-handler.jar,hbase-client.jar,hbase-common.jar,hbase-hadoop-compact.jar,hbase-hadoop2-compact.jar,hbase-protocol.jar,hbase-server.jar,metrics-core.jar,guava.jar --driver-class-path postgresql.jar --master yarn-client --files /usr/hdp/current/hbase-client/conf/hbase-site.xml --class SparkJS --driver-class-path /etc/hbase/2.5.0.0-1245/0 spark-js-1.jar The issue is gone and I can do what I need to do.

shigeru_takehar · ‎12-01-2016

Thank you for the recommendation, but I would like to solve this issue first. We are using hdp 2.5. Previously, we used hdp 2.3 where I could not run Spark with Phoenix. Can hdp 2.5 allow us to use Phoenix in Spark 1.6.2?

shigeru_takehar · ‎12-01-2016

I have a Hive table that is integrated with HBase table. It works fine on Hive command line to see data; however, when I try to do the same in Spark Java code where create a dataframe object by select statement and call show method, I see the following message forever: 16/11/30 19:40:31 INFO ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15802d56675006a, negotiated timeout = 90000 16/11/30 19:40:31 INFO RegionSizeCalculator: Calculating region sizes for table "st_tbl_1". 16/11/30 19:41:19 INFO RpcRetryingCaller: Call exception, tries=10, retries=35, started=48332 ms ago, cancelled=false, msg= 16/11/30 19:41:40 INFO RpcRetryingCaller: Call exception, tries=11, retries=35, started=68473 ms ago, cancelled=false, msg= 16/11/30 19:42:00 INFO RpcRetryingCaller: Call exception, tries=12, retries=35, started=88545 ms ago, cancelled=false, msg= 16/11/30 19:42:20 INFO RpcRetryingCaller: Call exception, tries=13, retries=35, started=108742 ms ago, cancelled=false, msg=

shigeru_takehar · ‎02-04-2016

Where did you copy your jdbc driver for Sqoop action?

shigeru_takehar · ‎02-03-2016

As far as I see sqoop action above, I don't see hive-site.xml file. I guess you added it into lib directory in the deployment directory, which will keep Hive action from running and it gives you error something like hive-site.xml permssion error. You should add the hive-site.xml file in "Files" in the Sqoop action.

shigeru_takehar · ‎02-02-2016

When I added Hive action before the Sqoop action, I got hive-site.xml file permission error. To avoid this problem, delete lib/hive-site.xml file, and add this file to inside of Sqoop action as a file.

shigeru_takehar · ‎02-02-2016

When I tried to run an Oozie workflow that contains Hive and Sqoop actions, I had the same problem. In my case, I had lib directory where hive-site.xml exists for Sqoop action. After I move hive-site.xml to the parent directory of lib directory, which is the HDFS deployment directory and I add the hive-site.xml as a file to Sqoop action, the workflow works. By the way, this is for HDP 2.3.2.

shigeru_takehar · ‎02-02-2016

I have successfully run Hive action from Oozie Workflow. My simple Hive operation does: drop table test1; create table test1 as select * from A_BASE; Followings are the steps: 1: run su - oozie from SSH window. 2: run hdfs dfs -put /usr/hdp/2.3.2.0-2950/atlas/hook/hive/* /user/oozie/share/lib/lib_20151027124452/hive (assume HDP 2.3.2 used) 3: create a workflow that contains a Hive action. 4: add a property oozie.action.sharelib.for.hive = hive,hcatalog,sqoop in Oozie parameters. 5: create a hive script like above and upload it from "Script name" in the Hive action edit page. 6: Save the workflow. 7: Run it. 8: It should run.

shigeru_takehar · ‎01-29-2016

It looks like HDP 2.3.2 already has this patch.

shigeru_takehar · ‎01-29-2016

I also tested HiveContext so that Hive processing works in Spark memory. It works.

Online	Offline
Last Visited	‎04-21-2017 07:40 PM

Member Since	‎12-21-2015 02:26 AM
Last Visited	‎04-21-2017 07:40 PM
Posts	43
Kudos received	10

Cloudera Community

Re: Java Spark Program and Hive table backed by HB...

Re: How to run Spark job from Oozie Workflow on HD...

Re: Oozie Sqoop Action works for list-databases bu...

Re: Java Spark Program and Hive table backed by HB...

Re: Java Spark Program and Hive table backed by HB...

Java Spark Program and Hive table backed by HBase ...

Re: Oozie Sqoop action throwing java.io.IOExceptio...

Re: Oozie Sqoop action throwing java.io.IOExceptio...

Re: Oozie Sqoop Action works for list-databases bu...

Re: Oozie Hive action hive-site.xml permission den...

Re: Hive action in failing in Oozie

Re: How to run Spark job from Oozie Workflow on HD...

Re: How to run Spark job from Oozie Workflow on HD...