In my data I have 8 different schemas. I want to create 8 different data frame for them and save them in 8 different tables in hive. So far I created a super bean class which holds shared attributes and each bean class extends it. Based on the type attribute I created different objects. The problem is I am unable to save them in different data frame. Is there any way I can do that? Here is my code so far, which works fine for one schema. xmlData.foreachRDD(
new Function2<JavaRDD<String>, Time, Void>() {
public Void call(JavaRDD<String> rdd, Time time) {
HiveContext hiveContext = JavaHiveContext.getInstance(rdd.context());
// Convert RDD[String] to RDD[case class] to DataFrame
JavaRDD<JavaRow> rowRDD = rdd.map(new Function<String, JavaRow>() {
public JavaRow call(String line) throws Exception{
String[] fields = line.split("\\|");
//JavaRow is my super class
JavaRow record = null;
if(fields[2].trim().equalsIgnoreCase("CDR")){
record = new GPRSClass(fields[0], fields[1]);
}
if(fields[2].trim().equalsIgnoreCase("Activation")){
record = new GbPdpContextActivation(fields[0], fields[1], fields[2], fields[3]); }
return record;}});
DataFrame df;
df = hiveContext.createDataFrame(rowRDD, JavaRow.class);
df.toDF().registerTempTable("Consumer");
System.out.println(df.count()+" ************Record Recived************");
df = hiveContext.createDataFrame(rowRDD, GPRSClass.class);
hiveContext.sql("CREATE TABLE if not exists gprs_data ( processor string, fileName string, type string, version string, id string )STORED AS ORC ");
df.save("/apps/hive/warehouse/data", "org.apache.spark.sql.hive.orc",SaveMode.Append);
}
return null; } });
... View more
