mapping data from spark into hive table

2 replies
2609 views
Translate All

Spark is getting data with a header showing col's A,C,D,B and the data under it. Next day we get the same data like this Col's B,D,A,C and then the next day we get data like col's A,C,B,D and so on randomly. Now we have to put this data in a hive table with col's as A,B,C,D. Can any one suggest me an idea how to write this script in spark?

Shu_ashuAnswer

Gold ⭐️⭐️⭐️

@Satya G

Read the CSV file with header as described here:

https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader

Once you are able to read the csv file with header then use .select method and select the col's as

#pyspark:

df= spark.read.csv(<file>).option("header", "true") //read the csv with header df1=df.select("A","B","C","D") //select the columns in an order df1.write.mode("<overwrite/append>").saveAsTable("<db_name>.<tab_name>")

Shu_ashuAnswer

Gold ⭐️⭐️⭐️

@Satya G

Read the CSV file with header as described here:

https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader

Once you are able to read the csv file with header then use .select method and select the col's as

#pyspark:

df= spark.read.csv(<file>).option("header", "true") //read the csv with header df1=df.select("A","B","C","D") //select the columns in an order df1.write.mode("<overwrite/append>").saveAsTable("<db_name>.<tab_name>")

Translate

gsatya0825Author

Silver ⭐️

@Shu Thank you.

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.