Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase error when I'm trying to store dataframes from an external Java Api into Phoenix in the sandbox.

HBase error when I'm trying to store dataframes from an external Java Api into Phoenix in the sandbox.

New Contributor

Hi all,

I obtain the next error when I execute my java code in my machine . My code creates Dataframes with Spark and store these dataframes in Phoenix(Phoenix is running in the sandbox).

java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:Fri Jan 06 13:52:02 CET 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68468: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=sandbox.hortonworks.com,16020,1483698935749, seqNum=0

(My java code is running outside of the VM)

I have set up the sandbox following the steps to be able to use Phoenix as is described in the Hortonworks tutorials (I activated phoenix and HBase using Anbari). I red that the port 16020 was closed and I opened it, but it continues without working.

In the VM everything is working properly, as I could see HMaster and HRegionServer are working.

The code I did in my java api is pretty simple,I just converted a RDD to a Dataframe with Booking objects and the write this dataframe in Phoenix giving the parameters specified because I was not able to use the function saveToPhoneix():

final JavaSparkContext sc = obtainContext();//obtaincontext returns javasparkcontext
SQLContext spark = new SQLContext(sc);
JavaRDD<Booking>  rawBooks = rawEvents.map(new BFunction());
DataFrame bookingDF = spark.createDataFrame(rawBooks, Booking.class);

bookingDF.write().format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(ImmutableMap.of("table", "B","zkUrl", "127.0.0.1:2181:/hbase-unsecure")).save();

And in my pom.xml I have the next :

<dependencies>
   <dependency>
      <groupId>org.apache.phoenix</groupId>
      <artifactId>phoenix-spark</artifactId>
      <version>4.9.0-HBase-1.2</version>
      <exclusions>
         <exclusion>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>servlet-api-2.5</artifactId>
         </exclusion>
         <exclusion>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jsp-2.1</artifactId>
         </exclusion>
         <exclusion>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jsp-api-2.1</artifactId>
         </exclusion>
         <exclusion>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty</artifactId>
         </exclusion>
         <exclusion>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty-util</artifactId>
         </exclusion>
         <exclusion>
            <groupId>javax.servlet.jsp</groupId>
            <artifactId>jsp-api</artifactId>
         </exclusion>
         <exclusion>
            <groupId>javax.servlet</groupId>
            <artifactId>servlet-api</artifactId>
         </exclusion>
      </exclusions>
   </dependency>

   <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.6.2</version>
   </dependency>

   <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.10</artifactId>
      <version>1.6.2</version>
   </dependency>

Could someone please give me a hint? I'm quite new using Hortonworks and I'm here stacked for some days and I don't understand what I'm doing wrong. I was thinking to increase the RPC time in hbase site xml file,but I don't think it is the problem.

Thanks a lot.

4 REPLIES 4

Re: HBase error when I'm trying to store dataframes from an external Java Api into Phoenix in the sandbox.

@daniel gerald orbegoso barrantes

How and where are you running the java code?

Can you check to see if you have the following host mapped in your hosts file where you are running the code: sandbox.hortonworks.com

Check out @Dan Zaratsian github examples on spark with phoenix. They are in scala but you can see the command line he uses in the header comments:

https://github.com/zaratsian/SparkPhoenix/blob/master/src/main/scala/com/github/zaratsian/SparkPhoen...

You might want to try Dan's example just to make sure your setup is correct.

Re: HBase error when I'm trying to store dataframes from an external Java Api into Phoenix in the sandbox.

New Contributor
@cduby

thanks for you answer, I'm running my java code using IntelliJ and yes I have this host mapped in my hosts file: /etc/hosts

127.0.0.1 sandbox.hortonworks.com I have checked the example you sent me and he is using the same parameters as I did. I read on the weekend this link where the guys from Hortonworks explain hot to set up Phoenix but it didn't help either.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/validat...

Re: HBase error when I'm trying to store dataframes from an external Java Api into Phoenix in the sandbox.

Super Collaborator

I see a couple network setup problems:

1. The exception says that it was unable to connect to sandbox.hortonworks.com. As far as I understand it should be the VM ip. Did you add it to the hosts file on your host?

2. In your code you are using 127.0.0.1 which would work on the sandbox VM, but you need to replace it with VM ip if you want to access it outside of the VM.

Re: HBase error when I'm trying to store dataframes from an external Java Api into Phoenix in the sandbox.

New Contributor

@Sergey Soldatov thanks for your answer. Yes I have this host mapped in my hosts file: /etc/hosts

127.0.0.1 sandbox.hortonworks.com The ports are forwarded from the VM to the Host and as I understood 127.0.0.1 is the VM ip therefore if I want to access from outside of the VM to the VM I have to use this IP.

https://community.hortonworks.com/questions/14136/how-do-i-connect-to-hbase-on-sandbox-vm-from-host....