@Vitor Batista Data Frames are supposed to be faster than Python RDD operations, check slide 20 of this presentation: http://www.slideshare.net/databricks/spark-summit-eu-2015-spark-dataframes-simple-and-fast-analysis-of-structured-data Could you try code below and check if it's faster? from pyspark.sql import SQLContext, Row
input_file = "hdfs:///tmp/your_text_file"
raw_rdd = sc.textFile(input_file)
csv_rdd = raw_rdd.map(lambda x: x.split(","))
row_data = csv_rdd.map(lambda p: Row(
field1=p[0],
field2=p[1],
field3=p[2]
)
)
df = sqlContext.createDataFrame(row_data)
categories = {}
idxCategories = [0,1,2]
for i in idxCategories: ##idxCategories contains indexes of rows that contains categorical data
distinctValues = df.map(lambda x : x[i]).distinct().collect()
categories[i] = distinctValues
print categories[0]
print categories[1]
print categories[2]
... View more
