Member since
06-11-2019
2
Posts
0
Kudos Received
0
Solutions
06-24-2019
05:49 AM
Hi All, With reference to below mentioned link, I need to include "phoenix-5.0.0.3.0.1.0-187-spark2.jar" while running spark jobs in order to communicate with phoenix. I tried searching this in official maven repository, but unable to find it. It is not present in phoenix folder in HDP. Does anyone have any link where I can find it. https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/phoenix-data-access/content/considerations_for_setting_up_spark.html
... View more
Labels:
06-11-2019
06:40 PM
I'm trying to read data from phoenix using pyspark. I'm getting below mentioned error: 19/06/11 17:20:12 WARN ConnectionImplementation: Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:527) at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:287) I'm loading the pyspark as follows: pyspark --name elasticsearch_sink --jars jars/phoenix-5.0.0.3.0.1.0-187-client.jar,jars/spark-sql-kafka-0-10_2.11-2.3.1.jar,jars/spark-streaming-kafka-0-10-assembly_2.11-2.3.1.jar,jars/elasticsearch-hadoop-6.8.0.jar --files "/home/user/spark/artifacts/jaas.conf,./user.keytab" --driver-java-options '-Djava.security.auth.login.config=/home/user/spark/artifacts/jaas.conf' --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=jaas.conf" --master yarn I'm running below mentioned code: from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.session import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *
from random import randrange
import json
sc = SparkContext.getOrCreate()
spark = SparkSession.builder.appName("es_sink").getOrCreate()
sqlc = SQLContext(sc)
df = sqlc.read.format("org.apache.phoenix.spark").option("zkUrl","zookeeper:2181").option("table","proxy").load()
... View more
Labels: