https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-convert-a-DataFrame-to-a-Vector-dense-in-scala/m-p/148297#M28346 <P>Perfect Solution.</P> Sat, 19 Nov 2016 09:29:25 GMT dqzhangchn 2016-11-19T09:29:25Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-convert-a-DataFrame-to-a-Vector-dense-in-scala/m-p/148295#M28344 <P>How to convert a DataFrame to a Vector.dense in scala</P><P>import org.apache.spark.mllib.linalg.Vectors import org.apache.spark.sql.functions.{concat, lit}</P><P>val f = bh.select($"GL20".alias("LABEL"), concat($"SYMBOL", lit(":"), $"DATE").alias("ID"), $"TIR", $"UO", $"ROC20" ) f.show(3)</P><P>+---------+------------------+--------+--------+--------+ | LABEL| ID| TIR| UO| ROC20| +---------+------------------+--------+--------+--------+ |-5.452071|DJI.IDX:2010-04-20|73.26948|65.55433| 3.0704| |-5.065461|DJT.IDX:2010-04-20|78.73316|68.14407|6.275064| |-6.747381|NDX.IDX:2010-04-20|77.02333|68.68713|3.796183| +---------+------------------+--------+--------+--------+</P><P>I want a new dataFrame in the format from the bh dataFrame above. </P><P>+------------------+--------------------+ | ID| FEATURES| +------------------+--------------------+ |DJI.IDX:2010-04-20|[73.26948,65.5543...| |DJT.IDX:2010-04-20|[78.73316,68.1440...| |NDX.IDX:2010-04-20|[77.02333,68.6871...| +------------------+--------------------+</P><P>If I hard code the values I can produce the above results but I need to get in programmatically from the bh dataFrame.</P><P>import org.apache.spark.ml.clustering.KMeans import org.apache.spark.mllib.linalg.Vectors</P><P>// Crates a DataFrame val df = sqlContext.createDataFrame(Seq( ("DJI.IDX:2010-04-20", Vectors.dense(73.26948, 65.55433, 3.0704)), ("DJT.IDX:2010-04-20", Vectors.dense(78.73316, 68.14407, 6.275064)), ("NDX.IDX:2010-04-20", Vectors.dense(77.02333, 68.68713, 3.796183)) )).toDF("ID", "FEATURES")</P><P>df.show()</P> Tue, 21 Apr 2026 13:30:21 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-convert-a-DataFrame-to-a-Vector-dense-in-scala/m-p/148295#M28344 wcarroll 2026-04-21T13:30:21Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-convert-a-DataFrame-to-a-Vector-dense-in-scala/m-p/148296#M28345 <P>Adding answer in case others need this. I used the VectorAssembler. </P><P><A href="http://spark.apache.org/docs/latest/ml-features.html#vectorassembler" target="_blank">http://spark.apache.org/docs/latest/ml-features.html#vectorassembler</A></P><P></P><P>val assembler = new VectorAssembler().setInputCols(Array("TIR", "UO", "ROC20" )).setOutputCol("FEATURES") val vd = assembler.transform(f)</P> Tue, 17 May 2016 17:10:38 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-convert-a-DataFrame-to-a-Vector-dense-in-scala/m-p/148296#M28345 wcarroll 2016-05-17T17:10:38Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-convert-a-DataFrame-to-a-Vector-dense-in-scala/m-p/148297#M28346 <P>Perfect Solution.</P> Sat, 19 Nov 2016 09:29:25 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-convert-a-DataFrame-to-a-Vector-dense-in-scala/m-p/148297#M28346 dqzhangchn 2016-11-19T09:29:25Z