Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark sort by key with descending order

Solved Go to solution
Highlighted

Spark sort by key with descending order

Expert Contributor

rdd.sortByKey() sorts in ascending order.

I want to sort in descending order.

I tried rdd.sortByKey("desc") but it did not work

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Spark sort by key with descending order

Try using rdd.sortByKey(false)

This will sort in descending order

2 REPLIES 2

Re: Spark sort by key with descending order

Try using rdd.sortByKey(false)

This will sort in descending order

Re: Spark sort by key with descending order

New Contributor

Try this code

from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
conf1 = SparkConf().setAppName('sort_desc')
sc1 = SparkContext(conf=conf1)
sql_context = SQLContext(sc1)
csv_file_path = 'emp.csv'
employee_rdd = sc1.textFile(csv_file_path).map(lambda line: line.split(','))
print(type(employee_rdd))
employee_rdd_sorted = employee_rdd.sortByKey(ascending= False)
employee_df = employee_rdd.toDF(['dept','ctc'])
employee_df_sorted = employee_rdd_sorted.toDF(['dept','ctc'])
Don't have an account?
Coming from Hortonworks? Activate your account here