question Re: mapping data from spark into hive table in Support Questions

mapping data from spark into hive table

gsatya0825 — Fri, 11 Jan 2019 14:22:44 GMT

Spark is getting data with a header showing col's A,C,D,B and the data under it. Next day we get the same data like this Col's B,D,A,C and then the next day we get data like col's A,C,B,D and so on randomly. Now we have to put this data in a hive table with col's as A,B,C,D. Can any one suggest me an idea how to write this script in spark?

Re: mapping data from spark into hive table

Shu_ashu — Sat, 12 Jan 2019 10:40:53 GMT

@Satya G

Read the CSV file with header as described here:

https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader

Once you are able to read the csv file with header then use .select method and select the col's as

#pyspark:

df= spark.read.csv(<file>).option("header", "true") //read the csv with header
df1=df.select("A","B","C","D") //select the columns in an order
df1.write.mode("<overwrite/append>").saveAsTable("<db_name>.<tab_name>")

Re: mapping data from spark into hive table

gsatya0825 — Mon, 14 Jan 2019 06:18:43 GMT

@Shu Thank you.