question Re: Spark Dataframes: How can I change the order of columns in Java/Scala? in Archives of Support Questions (Read Only)

Spark Dataframes: How can I change the order of columns in Java/Scala?

jestinm — Thu, 30 Jun 2016 02:31:24 GMT

After joining two dataframes, I find that the column order has changed what I supposed it would be.

Ex: Joining two data frames with columns [b,c,d,e] and [a,b] on b yields a column order of [b,a,c,d,e].

How can I change the order of the columns (e.g., [a,b,c,d,e])? I've found ways to do it in Python/R but not Scala or Java. Are there any methods that allow swapping or reordering of dataframe columns?

Re: Spark Dataframes: How can I change the order of columns in Java/Scala?

jyadav — Thu, 30 Jun 2016 02:38:36 GMT

Your sorting should happens on the basis of the key, here is an example for scala.

val file = sc.textFile("some_local_text_file_pathname")
val wordCounts = file.flatMap(line => line.split(" "))
  .map(word => (word, 1))
  .reduceByKey(_ + _, 1)  // 2nd arg configures one task (same as number of partitions)
  .map(item => item.swap) // interchanges position of entries in each tuple
  .sortByKey(true, 1) // 1st arg configures ascending sort, 2nd arg configures one task
  .map(item => item.swap)

Re: Spark Dataframes: How can I change the order of columns in Java/Scala?

psingh15 — Sat, 02 Jul 2016 20:07:25 GMT

@Jestin: Why do you need sorting columns in dataframes? Could u please elaborate.

However in Java there is no inbuilt function to reorder the columns.

Re: Spark Dataframes: How can I change the order of columns in Java/Scala?

TimothySpann — Sat, 02 Jul 2016 20:47:29 GMT

why does the order of columns matter?

Re: Spark Dataframes: How can I change the order of columns in Java/Scala?

psingh15 — Sat, 02 Jul 2016 22:29:16 GMT

There are scenarios(though bad) where data insertion requires the ordering of columns to be in Lexicographical Sorting while inserting data into db using JDBC connection. Not sure if jestin ma is facing similar issue.

Re: Spark Dataframes: How can I change the order of columns in Java/Scala?

don_jernigan — Sun, 03 Jul 2016 10:18:58 GMT

In order to reorder tuples (columns) in scala I think you just use a map like in Pyspark:

val rdd2 = rdd.map((x, y, z) => (z, y, x))

You should also be able to build key-value pairs this way too.

val rdd2 = rdd.map((x, y, z) => (z, (y, x)))

This is very handy if you want to follow it up with sortByKey().

Re: Spark Dataframes: How can I change the order of columns in Java/Scala?

venki2404 — Fri, 15 Jul 2016 23:39:33 GMT

All you need to do is use select (worked for me). Do the following:

val new_df = df.select("a", "b", "c", "d", "e") // Assuming you want a, b, c, d, e to be your order

@venki2404

All you need to do do is use select (worked for me). Do the following:

val new_df = df.select("a", "b", "c", "d", "e") // assuming the column order you need is a, b, c, d, e