Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issues with Spark On Hbase Connector

Issues with Spark On Hbase Connector

New Contributor

Regarding hbase connector https://github.com/hortonworks-spark/shc on CDH Cluster, as these connector changes are not yet bundled in CDH realeases, I am passing connector jar externally to spark job. If anyone already tried, can you please suggest

1. What versions of Hbase & Spark expected? I could not run examples provided using spark 1.6.0 & hbase 1.2.0
2. I get error when i run example provided here , any pointers on what i am doing wrong?

looks like spark not reading hbase-site.xml, but passed it in --files while spark-shell
e.g --files /etc/hbase/conf/hbase-site.xml,/etc/hbase/conf/hdfs-site.xml,/etc/hbase/conf/core-site.xml

error
16/08/27 12:35:00 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

3 REPLIES 3
Highlighted

Re: Issues with Spark On Hbase Connector

Contributor

Try passing the conf files as below

 

--conf spark.driver.extraClassPath=<your config path here> --conf spark.executor.extraClassPath=<your config path here>

Re: Issues with Spark On Hbase Connector

New Contributor

That doesn't work Umesh.

 

spark-shell --master yarn-client --num-executors 2 --driver-memory 512m --executor-memory 512m --executor-cores 1 --jars /home/sp09832/spark/jars/shc-0.0.11-1.6.1-s_2.10.jar --conf spark.driver.extraClassPath=/etc/hbase/conf/hbase-site.xml:/etc/hbase/conf/hdfs-site.xml:/etc/hbase/conf/core-site.xml --conf spark.executor.extraClassPath=/etc/hbase/conf/hbase-site.xml:/etc/hbase/conf/hdfs-site.xml:/etc/hbase/conf/core-site.xml

Re: Issues with Spark On Hbase Connector

Contributor

Well, it should work, but you can try --driver-class-path as you are using yarn-client mode and for sure Cloudera has a simmiler implementation. Please check here and this blog

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here