@Pedro Rodgers you can get your multiple files into a Spark RDD with: val data = sc.textFile("/user/pedro/pig_files/*txt") or even val data = sc.textFile("/user/pedro/pig_files") From this point onwards the Spark RDD 'data' will have as many partitions as there are pig files. Spark is just as happy with that, since distributing the data brings more speed and performance to anything you want to do on that RDD. Now if you want to merge those files into one and rewrite to HDFS again, it is just: data.repartition(1).saveAsTextFile("/user/pedro/new_file_dir") You can not determine the name of the output file (easily), just the HDFS folder will do Hope this helps ... View more

scala> val a = sc.textFile("/user/.../path/to/your/file").map(x => x.split("\t")).filter(x => x(0) != x(1)) scala> a.take(4) res2: Array[Array[String]] = Array(Array(1, 4), Array(2, 5), Array(1, 5)) Try the snippet above, just insert the path to your file on hdfs. ... View more

B) Create a hive table. The Hive table should have all the columns stated in your hive2parquet.csv file. Assume (col1, col2, col3). Also assume your csv file is in /tmp dir inside HDFS. 1- Log into Hive and at hive command prompt and execute 2- and 3- and C) below; // create the hive table 2- create table temp_txt (col1 string,col2 string, col3 string) row format delimited fields terminated by ','; // load the hive table with hive2parquet.csv file 3- load data input ' /tmp/hive2parquet.csv' into table temp_text; // Insert from table 'temp_txt' to table 'table_parquet_file' C- insert into table table_parquet_file select * from temp_txt; ... View more

Hi Pedro, python API for Spark is still missing, however there is a git project with a higher level API on top of Spark GraphX called GraphFrames: (GraphFrames) . The project claims: "GraphX is to RDDs as GraphFrames are to DataFrames." I haven't worked with it, however a quick test of their samples with Spark 1.6.2 worked: Use pyspark like this: pyspark --packages graphframes:graphframes:0.2.0-spark1.6-s_2.10 or use zeppelin and add the dependencies to the interpreter configuration. Maybe this library has what you need. ... View more