Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while inserting data into Kudu table from HIve table using Pyspark

Highlighted

Error while inserting data into Kudu table from HIve table using Pyspark

New Contributor

Version Details:
----------------
Cloudera - CDH 5.14
Apache Impala -              impala-2.11.0+cdh5.14.0+0
Apache Kudu - kudu-1.6.0+cdh5.14.0+0
Apache Spark - spark-1.6.0+cdh5.14.0+537

Problem trying to solve:
------------------------
We are having a change records (delta records) in HDFS Hive table.
Trying to upsert these change records table which have updates and inserts into a Kudu table using PySpark 1.6.

Steps executed:
---------------
Case 1:
csv_df = hive_context.read.format("com.databricks.spark.csv").load("%s" % csv_file_path)
csv_df.write.format('org.apache.kudu.spark.kudu').option('kudu.master', kudu_master).option('kudu.table',kudu_table_name).mode("append").save()

Case 2:
In the second case we tried to create a temptable from the dataframe and tried to insert the same into the kudu table.
csv_df.registerTempTable('makt_kudu_temp')
sqlContext.sql("""insert into <makt_db_name>.<makt_table_name> select * from makt_kudu_temp""")


In both the cases we got the below error.
  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/hadoop1/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/spark/python/pyspark/sql/context.py", line 580, in sql
    return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
  File "/hadoop1/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/hadoop1/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/spark/python/pyspark/sql/utils.py", line 45, in deco
    return f(*a, **kw)
  File "/hadoop1/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o48.sql.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.kudu.mapreduce.KuduTableInputFormat
        at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:308)
        at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1$$anonfun$3.apply(ClientWrapper.scala:358)
        at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1$$anonfun$3.apply(ClientWrapper.scala:341)
        at scala.Option.map(Option.scala:145)
  

2. We are getting the error with "Upsert" also when we tried to perform the operation.(Is upsert do not run by sqlContext).


At last, We used CASTING also in each and every column of the table but that also did not help.

2 REPLIES 2

Re: Error while inserting data into Kudu table from HIve table using Pyspark

Contributor

AFAIK there isn't any official guidance for using PySpark with Kudu. That said, have you taken a look at this repo that has some info on using Kudu with PySpark?

Re: Error while inserting data into Kudu table from HIve table using Pyspark

New Contributor

Hi,

 

Did you able to resolve the issue..?  If yes, how did you do..i am also having the same issue..?

 

Thanks,