Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while loading dataframe into a hive partition

Error while loading dataframe into a hive partition

Contributor

Im trying to load a dataframe into hive table which is partitioned like below.

> create table emptab(id int, name String, salary int, dept String)
> partitioned by (location String)
> row format delimited
> fields terminated by ','
> stored as parquet;

 

I have a dataframe created in the below format:

val empfile = sc.textFile("emp")
val empdata = empfile.map(e => e.split(","))
case class employee(id:Int, name:String, salary:Int, dept:String)
val empRDD = empdata.map(e => employee(e(0).toInt, e(1), e(2).toint, e(3)))
val empDF = empRDD.toDF()
empDF.write.partitionBy("location").insertInto("/user/hive/warehouse/emptab/location=England")

But Im getting an error as below:

empDF.write.partitionBy("location").insertInto("/user/hive/warehouse/emptab/location=India")
java.lang.RuntimeException: [1.1] failure: identifier expected
/user/hive/warehouse/emptab/location=England

 

Data in my input file: emp

 

 ---+-------+------+-----+
| id|   name|salary| dept|
+---+-------+------+-----+
|  1|   Mark|  1000|   HR|
|  2|  Peter|  1200|SALES|
|  3|  Henry|  1500|   HR|
|  4|   Adam|  2000|   IT|
|  5|  Steve|  2500|   IT|
|  6|  Brian|  2700|   IT|
|  7|Michael|  3000|   HR|
|  8|  Steve| 10000|SALES|
|  9|  Peter|  7000|   HR|
| 10|    Dan|  6000|   BS|
+---+-------+------+-----+

Could anyone tell what is the mistake I am doing here and how can I correct it ?

1 REPLY 1
Highlighted

Re: Error while loading dataframe into a hive partition

New Contributor

If you're going to partition the data by location, shouldn't your dataframe contain a column named "location" that it can use to partition the data by?

 

I haven't implemented partitioning in Hive / Hadoop, but that's been true for every other database I've worked with.

 

Don't have an account?
Coming from Hortonworks? Activate your account here