Reply
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Problem with hbase classpath within s spark job (cloudera quickstart)

I'm having a simple spark job (done in java) that I'm executing on a cloudera quickstart plateform (learning phase) and I wanted to create a hbase table.

 

However hbase/zookeeper is not found:

 

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/ZooKeeperConnectionException
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:270)
           at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:319)
           at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
           at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ZooKeeperConnectionException
           at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
           at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
           at java.security.AccessController.doPrivileged(Native Method)
           at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

  

I tried to set the HBASE_CLASSPATH from the /usr/lib/hbase/conf/hbase-env.sh to /usr/lib/hbase/ but it didn't worked. 

My code:

 

	Configuration conf = HBaseConfiguration.create();
		
		try {
			HBaseAdmin admin = new HBaseAdmin(conf);
		} catch (MasterNotRunningException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (ZooKeeperConnectionException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf("people"));

 

What should I do?

 

Thank you!

 

GHERMAN Alina
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: Problem with hbase classpath within s spark job (cloudera quickstart)

I forgot to say that I also tried to add:

<property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>usr/local/zookeeper</value>
  </property>
  <property>
      <name>hbase.zookeeper.property.clientPort</name>
      <value>2181</value>
  </property>

 in my hbase-env.sh script (before that, in this script there was nothing concerning zookeeper)

 

 

Thank you!

GHERMAN Alina
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: Problem with hbase classpath within s spark job (cloudera quickstart)

And my pom is containg the folowing components:

 

	<dependency><!-- hbase -->
			<groupId>org.apache.hbase</groupId>
			<artifactId>hbase-server</artifactId>
			<version>${hbase.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency><!-- hbase -->
			<groupId>org.apache.hbase</groupId>
			<artifactId>hbase-common</artifactId>
			<version>${hbase.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency><!-- hbase -->
			<groupId>org.apache.hbase</groupId>
			<artifactId>hbase-client</artifactId>
			<version>${hbase.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency><!-- zookeeper -->
			<groupId>org.apache.zookeeper</groupId>
			<artifactId>zookeeper</artifactId>
			<version>${zookeeper.version}</version>
			<scope>provided</scope>
		</dependency>

 

GHERMAN Alina
Cloudera Employee
Posts: 576
Registered: ‎01-20-2014

Re: Problem with hbase classpath within s spark job (cloudera quickstart)

Could you please try setting:
$ export SPARK_CLASSPATH=$SPARK_CLASSPATH:`hbase classpath`

Ensure the value contains /etc/hbase/conf as well, otherwise append it to the list. Do let us know if this solved your problem
Regards,
Gautam Gopalakrishnan
Cloudera Support
New Contributor
Posts: 1
Registered: ‎07-20-2018

Re: Problem with hbase classpath within s spark job (cloudera quickstart)

It doesn't solve the problem. My HBASE configuration was in different directory other than /etc/hbase/conf but as you suggsted to append the SPARK_CLASSPATH with the HBASE configuration path. I did the same in my enviornment but it didn't work.

 

What I'm trying here - I've Spark2.3 R3 installed on Cloudera Platform and trying to connect HBASE which is running on different cluster. It's throwing an exception "hbase package is not a member of org.apache.hadoop package" while running Spark2.3

 

Can you please sugguest me how I pass the HBASE configuration/libraries to Spark2.3

Announcements