Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to access hive table with hbase as storage from sparkR

Unable to access hive table with hbase as storage from sparkR

Rising Star
17/05/22 21:01:11 ERROR RBackendHandler: dfToCols on org.apache.spark.sql.api.r.SQLUtils failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
  java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
        at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
        at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:89)
        at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculat
2 REPLIES 2
Highlighted

Re: Unable to access hive table with hbase as storage from sparkR

Rising Star

sparkR doesn't work only for hive table with hbase as storage. The hive managed table works as expected.

/usr/hdp/2.5.3.0-37/spark/bin/sparkR --master local \
> --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 \
> --repositories http://repo.hortonworks.com/content/groups/public/ \
> --conf "spark.executor.extraClassPath=/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar:/usr/hdp/2.5.3.0-37/spark/lib/spark-examples-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.7.0.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-common-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-client-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-server-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-protocol-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.5.3.0-37/hbase/lib/guava-12.0.1.jar:/etc/hbase/conf/hbase-site.xml" \
> --conf "spark.driver.extraClassPath=/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar:/usr/hdp/2.5.3.0-37/spark/lib/spark-examples-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.7.0.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-common-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-client-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-server-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-protocol-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.5.3.0-37/hbase/lib/guava-12.0.1.jar:/etc/hbase/conf/hbase-site.xml"
Highlighted

Re: Unable to access hive table with hbase as storage from sparkR

Rising Star

Got it working

Step 1 : yum install -y R R-devel libcurl-devel openssl-devel

Step 2: In spark context, Run install.packages("knitr")

Step 3 : ln -s /etc/hbase/conf/hbase-site.xml /etc/spark/conf/hbase-site.xml

Step 4:

/usr/hdp/2.5.3.0-37/spark/bin/sparkR --master local \
--packages com.hortonworks:shc:1.0.0-1.6-s_2.10 \
--repositories http://repo.hortonworks.com/content/groups/public/ \
--conf "spark.executor.extraClassPath=/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar:/usr/hdp/2.5.3.0-37/spark/lib/spark-examples-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.7.0.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-common-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-client-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-server-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-protocol-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.5.3.0-37/hbase/lib/guava-12.0.1.jar:/etc/hbase/conf/hbase-site.xml" \
--conf "spark.driver.extraClassPath=/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar:/usr/hdp/2.5.3.0-37/spark/lib/spark-examples-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.7.0.2.5.3.0-37.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-common-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-client-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-server-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/hbase-protocol-1.1.2.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.5.3.0-37/hbase/lib/guava-12.0.1.jar:/etc/hbase/conf/hbase-site.xml"
Don't have an account?
Coming from Hortonworks? Activate your account here