Created on 05-19-2016 01:07 AM - edited 09-16-2022 03:20 AM
We are running spark streaming to get the feed from Kafka. Now we are trying to use Phoenix JDBC to extract some data from HBASE. When I run the code in local its running fine without any issues, but when I run it using yarn-cluster it throws below exception. Using CDH 5.4.4.
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
Below is the code snippet:
def check(arg0:String,arg1:String)
{
val query = "query"
Class.forName("org.apache.phoenix.jdbc.PhoenixDriver");
var conn = DriverManager.getConnection("jdbc:phoenix:IP:zkport:/znode");
val statement = conn.createStatement()
val resultSet = statement.executeQuery(query)
var out="";
while(resultSet.next())
{
out=resultSet.getString("..")
}
}
And the SBT dependency added is
libraryDependencies += "org.apache.phoenix" % "phoenix-core" % "4.5.2-cdh5.4.5"
I manually checked for the missing class, and it is there in phoenix-core jar. What is the reason behind yarn/spark throwing an exception. I tried adding it as a separate classpath that doesn't work either. Any workaround for this.
Created 05-19-2016 06:59 AM
Adding phoenix-core to classpath.txt worked. But a weird issue.
Created 05-19-2016 05:47 AM
Hello,
are you using the Phonex Clab ? What is the version of it ?
Kind Regards
Created 05-19-2016 06:45 AM
Yes, 4.5.2-cdh5.4.5 is the one in my dependency. Got that from cloudera repo https://repository.cloudera.com/cloudera/cloudera-repos/org/apache/phoenix/phoenix-core/
Created 05-19-2016 06:59 AM
Adding phoenix-core to classpath.txt worked. But a weird issue.
Created 06-04-2016 11:10 PM
I am getting the same error when I try to load a phoenix table into Spark DataFrame. Have you come across same issue with dataframes?
Created 06-06-2016 07:09 AM
Not with dataframes but i think it is a collon issue with phoenix+spark streaming. You can try adding the jars to classpath and execute