Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Change master dataframe into another(child) dataframe while splitting dataframe I'm trying to give specified columns name using withColumn ans subString

Highlighted

Change master dataframe into another(child) dataframe while splitting dataframe I'm trying to give specified columns name using withColumn ans subString

New Contributor

I have a dataframe where i'm trying to split into another dataframe(or child dataframe) while splitting i'm trying give columns name using withColumn and withColumn i'm calling inside for loop.

ChDF=
JK059122016022DK165754000000  000100000 L000002600901100020000           00007908985600001050000000N    
JK059122016022DK100000000000  000200000 9080003543317400010000           00009390150200001410000000N    
JK059122016022DK160000000000  000100000 90800034921015000100000000000000000014600000000000000000000N    
JK059122016022DK165752000000  000100000 9080002837011800020000           00011110000000000000000000N    
JK059122016022DK100000000000  920161206 83824VS12240000000000000000000000000000000000020161206080204
JK059122016022DK165000000000  0001IDGDS S000093102555500020000           00004460000000000000000000N

objectList=List(Map(type->deCode1,value1->45,value2->1),Map(type->deCode2,value1->52,value2->6),Map(type->deCode3,value1->68,value2->7),Map(type->deCode4,value1->95,value2->2),Map(type->deCode5,value1->37,value2->8),Map(type->deCode6,value1->05,value2->8))
objectList.foreach(iteam => 
ChDF.withColumn(iteam("type"),substring(ChildDF("masterDF"),iteam("value1").asInstanceOf[Int],iteam("value2").asInstanceOf[Int]))
)
ChDF.createOrReplaceTempView("ChDFTable")
val queryDFTable = "SELECT * from ChDFTable"
sparkSession.sql(queryDFTable).cache().toDF().show()

Output coming from above code

       masterDF
JK059122016022DK165754000000 
JK059122016022DK100000000000 
JK059122016022DK160000000000 
JK059122016022DK165752000000 
JK059122016022DK100000000000 
JK059122016022DK165000000000

Expect output(Where JJJJJJJJJ is splitted values come from master dataframe)

      masterDF                   deCode1        deCode2       deCode3     deCode4    deCode5              
JK059122016022DK165754000000    JJJJJJJJJ      JJJJJJJJJ     JJJJJJJJJ   JJJJJJJJJ  JJJJJJJJJ  
JK059122016022DK100000000000    JJJJJJJJJ      JJJJJJJJJ     JJJJJJJJJ   JJJJJJJJJ  JJJJJJJJJ
JK059122016022DK160000000000    JJJJJJJJJ      JJJJJJJJJ     JJJJJJJJJ   JJJJJJJJJ  JJJJJJJJJ
JK059122016022DK165752000000    JJJJJJJJJ      JJJJJJJJJ     JJJJJJJJJ   JJJJJJJJJ  JJJJJJJJJ
JK059122016022DK100000000000    JJJJJJJJJ      JJJJJJJJJ     JJJJJJJJJ   JJJJJJJJJ  JJJJJJJJJ 
JK059122016022DK165000000000    JJJJJJJJJ      JJJJJJJJJ     JJJJJJJJJ   JJJJJJJJJ  JJJJJJJJJ
Don't have an account?
Coming from Hortonworks? Activate your account here