question Spark sort by key with descending order in Archives of Support Questions (Read Only)

Spark sort by key with descending order

sreeviswa_athic — Thu, 19 Oct 2017 10:15:53 GMT

rdd.sortByKey() sorts in ascending order.

I want to sort in descending order.

I tried rdd.sortByKey("desc") but it did not work

Re: Spark sort by key with descending order

dineshc — Thu, 19 Oct 2017 10:17:36 GMT

Try using rdd.sortByKey(false)

This will sort in descending order

Re: Spark sort by key with descending order

thangarajanp — Fri, 20 Oct 2017 13:34:20 GMT

Try this code

from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
conf1 = SparkConf().setAppName('sort_desc')
sc1 = SparkContext(conf=conf1)
sql_context = SQLContext(sc1)
csv_file_path = 'emp.csv'
employee_rdd = sc1.textFile(csv_file_path).map(lambda line: line.split(','))
print(type(employee_rdd))
employee_rdd_sorted = employee_rdd.sortByKey(ascending= False)
employee_df = employee_rdd.toDF(['dept','ctc'])
employee_df_sorted = employee_rdd_sorted.toDF(['dept','ctc'])