Hello, I am on CDH 5.7
I have an HBase table. To be able to access it from Impala/Hive, I have created a Hive external table pointing to it, using HBaseStorageHandler, using the following approach:
CREATE EXTERNAL TABLE hbase_utenti(key string, val1 string, ... , valX string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:a,cf1:b, ... ,cfX:X")
TBLPROPERTIES("hbase.table.name" = "utenti");This WORKS smoothly, I can query the table from Impala/Hive
Now I have the following PySpark script that I want to run via Spark Action (just an excerpt):
## IMPORT FUNCTIONS
import sys
import getopt
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import *
from pyspark.sql.types import *
## CREATE SQLCONTEXT
sconf = SparkConf().setAppName("ChurnVoip").setMaster("yarn-cluster").set("spark.driver.memory", "3g").set("spark.executor.memory", "1g").set("spark.executor.cores", 2)
sc = SparkContext(conf=sconf)
sqlContext = HiveContext(sc)
## CREATE MAIN DATAFRAME
hbase_utenti_DF = sqlContext.table("msgnet.hbase_utenti")
## SAVE HBASE TABLE AS "REAL" HIVE TABLE
hbase_utenti_filtered_DF = hbase_utenti_DF \
.select(hbase_utenti_DF["USERID"].alias("HHUTE_USERID"), \
hbase_utenti_DF["RIVENDITORE"].alias("HHUTE_RIVENDITORE"), \
hbase_utenti_DF["DATA_COLL"].alias("HHUTE_DATA_COLL"), \
hbase_utenti_DF["CREDITO"].alias("HHUTE_CREDITO"), \
hbase_utenti_DF["VIRTUAL"].alias("HHUTE_VIRTUAL"), \
hbase_utenti_DF["POST"].alias("HHUTE_POST")) \
.filter("((HHUTE_RIVENDITORE = 1) and \
(HHUTE_DATA_COLL='2016-12-30T00:00:00Z'))") \
.saveAsTable('msgnet.analisi_voipout_utenti_credito_2016_rif')Please note that if I run the above Python script via "spark-submit" OR if I run Pyspark shell and copy-paste the code into it, IT WORKS smoothly
Problem arise when I try to schedule an Oozie Spark Action doing the job. In this case, I get the following exception:
...
2017-02-05 10:47:52,106 ERROR [Thread-8] hive.log (MetaStoreUtils.java:getDeserializer(387)) - error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
...
hbase_utenti_DF = sqlContext.table("msgnet.hbase_utenti")
File "/hdp01/yarn/nm/usercache/msgnet/appcache/application_1486053545979_0053/container_1486053545979_0053_02_000001/pyspark.zip/pyspark/sql/context.py", line 593, in table
---
py4j.protocol.Py4JJavaError: An error occurred while calling o56.table.
: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandlerNow, I'm pretty sure the whole problem boils down to Jars not being included. But I've tried and tried to add Paths and Variables to my Oozie Spark Action, with no success. What I've tried to do includes the following:
- Adding the following "Hadoop Properties" to my Workflow Settings (gear icon on top right of the Workflow in Hue):
spark.executor.extraClassPath /opt/cloudera/parcels/CDH/lib/hive/lib/*
spark.executor.extraClassPath /opt/cloudera/parcels/CDH/lib/hbase/lib/*
- Adding the following to my Spark Action's
--files
hdfs:///user/<XXX>/hive-site.xml,
hdfs:///user/<XXX>/hive-hbase-handler-1.1.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-client-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-server-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-hadoop-compat-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-hadoop2-compat-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/metrics-core-2.2.0.jar
--driver-class-path /etc/hbase/conf:/opt/cloudera/parcels/CDH/lib/hbase/lib/*
Can anybody please shed some light on what I'm missing here? I'm really out of ideas, after trying and trying.
Thanks for any insight!
