Member since
02-20-2020
2
Posts
0
Kudos Received
0
Solutions
02-18-2021
06:03 AM
Please add kudu-spark2-tools_2.11-1.6.0.jar depending on your spark and scala version. Use spark context instead of hive. Apache Spark 2.3 Below is the code for your reference: ------------------------------------- Read kudu table from pyspark with below code: kuduDF = spark.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"IP of master").option('kudu.table',"impala::TABLE: name").load() kuduDF.show(5) Write to kudu table with below code: DF.write.format('org.apache.kudu.spark.kudu').option('kudu.master',"IP of master").option('kudu.table',"impala::TABLE: name").mode("append").save() ---------------------------------------- Reference link: https://medium.com/@sciencecommitter/how-to-read-from-and-write-to-kudu-tables-in-pyspark-via-impala... If in case you want to use Scala below is the reference link: https://kudu.apache.org/docs/developing.html
... View more
02-20-2020
11:45 PM
Use spark context instead of hive. Apache Spark 2.3 Below is the code for your reference: ------------------------------------- Read kudu table from pyspark with below code: kuduDF = spark.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"IP of master").option('kudu.table',"impala::TABLE: name").load() kuduDF.show(5) Write to kudu table with below code: DF.write.format('org.apache.kudu.spark.kudu').option('kudu.master',"IP of master").option('kudu.table',"impala::TABLE: name").mode("append").save() ---------------------------------------- Reference link: https://medium.com/@sciencecommitter/how-to-read-from-and-write-to-kudu-tables-in-pyspark-via-impala-c4334b98cf05 If in case you want to use Scala below is the reference link: https://kudu.apache.org/docs/developing.html
... View more