mapping data from spark into hive table

[#1]

Spark is getting data with a header showing col's A,C,D,B and the data under it. Next day we get the same data like this Col's B,D,A,C and then the next day we get data like col's A,C,B,D and so on randomly. Now we have to put this data in a hive table with col's as A,B,C,D. Can any one suggest me an idea how to write this script in spark?

[#2 · Accepted solution]

@Satya G

Read the CSV file with header as described here:

https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader

Once you are able to read the csv file with header then use .select method and select the col's as

#pyspark:

df= spark.read.csv(<file>).option("header", "true") //read the csv with header
df1=df.select("A","B","C","D") //select the columns in an order
df1.write.mode("<overwrite/append>").saveAsTable("<db_name>.<tab_name>")

[#3]

@Shu Thank you.