question Re: Apache phoenix 4.5.1 with spark 1.3.1 Classnotfoundexception in Archives of Support Questions (Read Only)

Apache phoenix 4.5.1 with spark 1.3.1 Classnotfoundexception

vickyiv — Fri, 16 Sep 2022 10:20:41 GMT

We are running spark streaming to get the feed from Kafka. Now we are trying to use Phoenix JDBC to extract some data from HBASE. When I run the code in local its running fine without any issues, but when I run it using yarn-cluster it throws below exception. Using CDH 5.4.4.

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:190)

Below is the code snippet:

def check(arg0:String,arg1:String)
{
val query = "query"

        Class.forName("org.apache.phoenix.jdbc.PhoenixDriver");
    var conn = DriverManager.getConnection("jdbc:phoenix:IP:zkport:/znode");
        val statement = conn.createStatement()
        val resultSet = statement.executeQuery(query)
        var out="";
        while(resultSet.next())
        {
          out=resultSet.getString("..")
        }
}

And the SBT dependency added is

libraryDependencies += "org.apache.phoenix" % "phoenix-core" % "4.5.2-cdh5.4.5"

I manually checked for the missing class, and it is there in phoenix-core jar. What is the reason behind yarn/spark throwing an exception. I tried adding it as a separate classpath that doesn't work either. Any workaround for this.

Re: Apache phoenix 4.5.1 with spark 1.3.1 Classnotfoundexception

matdba — Thu, 19 May 2016 12:47:41 GMT

Hello,

are you using the Phonex Clab ? What is the version of it ?

Kind Regards

Re: Apache phoenix 4.5.1 with spark 1.3.1 Classnotfoundexception

vickyiv — Thu, 19 May 2016 13:45:02 GMT

Yes, 4.5.2-cdh5.4.5 is the one in my dependency. Got that from cloudera repo https://repository.cloudera.com/cloudera/cloudera-repos/org/apache/phoenix/phoenix-core/

Re: Apache phoenix 4.5.1 with spark 1.3.1 Classnotfoundexception

vickyiv — Thu, 19 May 2016 13:59:53 GMT

Adding phoenix-core to classpath.txt worked. But a weird issue.

Re: Apache phoenix 4.5.1 with spark 1.3.1 Classnotfoundexception

tsyed — Sun, 05 Jun 2016 06:10:28 GMT

I am getting the same error when I try to load a phoenix table into Spark DataFrame. Have you come across same issue with dataframes?

Re: Apache phoenix 4.5.1 with spark 1.3.1 Classnotfoundexception

vickyiv — Mon, 06 Jun 2016 14:09:43 GMT

Not with dataframes but i think it is a collon issue with phoenix+spark streaming. You can try adding the jars to classpath and execute