Member since
09-07-2018
7
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
12001 | 11-21-2018 01:54 AM |
11-21-2018
01:54 AM
Thank you! I have change it. Also I have changed my paths. Because the path is for directory and not for a file. I have also added a / to my path. Now I get results which I have expected. I changed "setMaster to "local" because it is just a small Cloudera VM without cluster. This is a simple Spark script which can be executed in hue per Spark editor: from pyspark import SparkContext, SparkConf appNameTEST ="my first working application" conf = SparkConf().setAppName(appNameTEST).setMaster("local") sc = SparkContext(conf=conf) text_file = sc.textFile("hdfs:///user/hive/warehouse/TEST/FilePath") counts = text_file.flatMap(lambda line: line.split(" ")) \ .map(lambda word: (word, 1)) \ .reduceByKey(lambda a, b: a + b) counts.saveAsTextFile("hdfs:///user/hive/warehouse/TEST/RESULT")
... View more